Torben's Wiki - User contributions [en]

Pnpm Node Package Manager

2026-04-18T18:40:45Z

Torben:

==Commands==
# install packages locally
pnpm install
# add package
pnpm add vue
# add dev package
pnpm add -D vitest

# update all packages to latest minor version
pnpm up
# update all packages to latest version
pnpm up --latest

# run package
pnpm exec vitest --watch=false

# run script (defined in package.json)
pnpm run myscript

# run without installing to repo
pnpx cowsay Moin

# check used packages for vulnerabilities
pnpm audit
pnpm audit --fix
pnpm install

AI LLM for Coding

2026-04-18T18:24:38Z

Torben:

==Mistral Devstral==
uv tool install mistral-vibe
# run via
vibe

Obtain API key from [https://console.mistral.ai/codestral/cli]

==Google Gemini==
pros: free to use with google account
create your API key at https://aistudio.google.com/app/api-keys

Install
npm install -g @google/gemini-cli@latest
gemini
# or just run
npx https://github.com/google-gemini/gemini-cli
usage
cd /tmp
export GEMINI_API_KEY="xxx"
git clone https://github.com/entorb/meeting-meter
cd myProject
gemini

to switch model from pro to flash:
export GEMINI_MODEL="gemini-2.5-flash"

secret alternative: .env file
GEMINI_API_KEY=xxx

==Google Antigravity==
https://antigravity.google/download

==Claude==
===Web===
https://claude.ai 
pro: Daily reset free quota 
cons: need to manually upload single files

===Console===
Claude Console via API and VISA Pre-Paid: https://console.anthropic.com 
pro: fully integrated in code base 
cons: costs money, 5€ are easily spend
npm install -g @anthropic-ai/claude-code
cd myProject
claude

==Tips==
* E2E tests (e.g. Cypress) are very important
* Use tools for code formatting, linting, type-hints
* Configure these tools to produce minimum output to reduce token consumption. e.g. for
** Prettier: --log-level silent
** Cypress: cypress run --e2e --quiet
* For Refactoring/renaming: better do it manually, via vscode, that is a lot faster

===Cavemen style===
https://github.com/JuliusBrussee/caveman
npx skills add JuliusBrussee/caveman -a github-copilot
It has an option to install to local home dir instead of to project repo.

AI LLM for Coding

2026-04-18T16:37:49Z

Torben: /* Mistral Devstral */

==Mistral Devstral==
uv tool install mistral-vibe
Obtain API key from [https://console.mistral.ai/codestral/cli]

==Google Gemini==
pros: free to use with google account
create your API key at https://aistudio.google.com/app/api-keys

Install
npm install -g @google/gemini-cli@latest
gemini
# or just run
npx https://github.com/google-gemini/gemini-cli
usage
cd /tmp
export GEMINI_API_KEY="xxx"
git clone https://github.com/entorb/meeting-meter
cd myProject
gemini

to switch model from pro to flash:
export GEMINI_MODEL="gemini-2.5-flash"

secret alternative: .env file
GEMINI_API_KEY=xxx

==Google Antigravity==
https://antigravity.google/download

==Claude==
===Web===
https://claude.ai 
pro: Daily reset free quota 
cons: need to manually upload single files

===Console===
Claude Console via API and VISA Pre-Paid: https://console.anthropic.com 
pro: fully integrated in code base 
cons: costs money, 5€ are easily spend
npm install -g @anthropic-ai/claude-code
cd myProject
claude

==Tips==
* E2E tests (e.g. Cypress) are very important
* Use tools for code formatting, linting, type-hints
* Configure these tools to produce minimum output to reduce token consumption. e.g. for
** Prettier: --log-level silent
** Cypress: cypress run --e2e --quiet
* For Refactoring/renaming: better do it manually, via vscode, that is a lot faster

===Cavemen style===
https://github.com/JuliusBrussee/caveman
npx skills add JuliusBrussee/caveman -a github-copilot
It has an option to install to local home dir instead of to project repo.

AI LLM for Coding

2026-04-18T16:18:50Z

Torben:

==Mistral Devstral==
uv tool install mistral-vibe

==Google Gemini==
pros: free to use with google account
create your API key at https://aistudio.google.com/app/api-keys

Install
npm install -g @google/gemini-cli@latest
gemini
# or just run
npx https://github.com/google-gemini/gemini-cli
usage
cd /tmp
export GEMINI_API_KEY="xxx"
git clone https://github.com/entorb/meeting-meter
cd myProject
gemini

to switch model from pro to flash:
export GEMINI_MODEL="gemini-2.5-flash"

secret alternative: .env file
GEMINI_API_KEY=xxx

==Google Antigravity==
https://antigravity.google/download

==Claude==
===Web===
https://claude.ai 
pro: Daily reset free quota 
cons: need to manually upload single files

===Console===
Claude Console via API and VISA Pre-Paid: https://console.anthropic.com 
pro: fully integrated in code base 
cons: costs money, 5€ are easily spend
npm install -g @anthropic-ai/claude-code
cd myProject
claude

==Tips==
* E2E tests (e.g. Cypress) are very important
* Use tools for code formatting, linting, type-hints
* Configure these tools to produce minimum output to reduce token consumption. e.g. for
** Prettier: --log-level silent
** Cypress: cypress run --e2e --quiet
* For Refactoring/renaming: better do it manually, via vscode, that is a lot faster

===Cavemen style===
https://github.com/JuliusBrussee/caveman
npx skills add JuliusBrussee/caveman -a github-copilot
It has an option to install to local home dir instead of to project repo.

Pnpm Node Package Manager

2026-04-18T16:12:11Z

Torben: Created page with "==Commands== # install packages locally pnpm install # add package pnpm add vue # add dev package pnpm add -D vitest # update pnpm up # run package pnpm exec vitest --watch=false # run script (defined in package.json) pnpm run myscript # check used packages for vulnerabilities pnpm audit pnpm audit --fix pnpm install"

==Commands==
# install packages locally
pnpm install
# add package
pnpm add vue
# add dev package
pnpm add -D vitest

# update
pnpm up

# run package
pnpm exec vitest --watch=false

# run script (defined in package.json)
pnpm run myscript

# check used packages for vulnerabilities
pnpm audit
pnpm audit --fix
pnpm install

Mac

2026-04-18T16:04:13Z

Torben:

[[Category:Software]][[Category:Apple]]
==Terminal==
===Set Hostname===
from [https://www.autodesk.com/support/technical/article/caas/sfdcarticles/sfdcarticles/Setting-the-Mac-hostname-or-computer-name-from-the-terminal.html]
sudo scutil --set HostName T-MB-2.fritz.box
sudo scutil --set LocalHostName T-MB-2
sudo scutil --set ComputerName T-MB-2
dscacheutil -flushcache

==Shortcuts==
Control + K : fill 2nd clipboad by cut text from cursor till end of line
Control + Y : paste from 2nd clipboad
Finder / Open file Dialogs
Command + Shift + . : show hidden files
Windows Management
(better use Rectangle App, see below)
Control + Tab : Switch between applications
Control + < : Switch between windows of same application
swipe 3-finger-upwards : Mission Control
Emoji
Control+Command+Space

==Disabling/speedup animations==
# Window opening animations
defaults write NSGlobalDomain NSAutomaticWindowAnimationsEnabled -bool true
# revert
# defaults write NSGlobalDomain

# The resizing animation
defaults write NSGlobalDomain NSWindowResizeTime -float 0.001
# revert
# defaults write NSGlobalDomain NSWindowResizeTime -float 0.2

# The Quick Look window animation
defaults write -g QLPanelAnimationDuration -float 0
# revert
# defaults delete -g QLPanelAnimationDuration

# Launching an app from the Dock
defaults write com.apple.dock launchanim -bool false
# revert
# defaults write com.apple.dock launchanim -bool true

==Time Machine Backups==
===Prepare external Hard Disk===
Formating problem
Erase process has failed, press done to continue. Mediakit reports not enough space on device for requested operation.
Quoting [https://www.reddit.com/r/applehelp/comments/40yvjh/disk_utility_fails_to_eraseformat_an_external_hdd/] :
Here's what I would try. First run
diskutil list
to get the name to the disk you're trying to format. The below commands assume this is "disk1", but replace "disk1" with the correct disk if it's something different.
Now unmount the disk:
diskutil unmountDisk force disk1
and then write zeros to the boot sector:
sudo dd if=/dev/zero of=/dev/disk1 bs=1024 count=1024
finally attempt to partition it again:
diskutil partitionDisk disk1 GPT JHFS+ "My External HD" 0g

===Encryption Takes For Ever===
Quoting [https://discussions.apple.com/thread/7813232]: To speed up the process of TM encryption, go to Disk Utility, select your external HDD and erase and use the option journaled/encrypted. Once your external HDD is erased, you can redo the backup and the encryption will be as fast as a flash.

===Delete old backups===
mount backup drive in filder via connect to server
afp://my_DS_Hostname
now it can be found under Locations and you can delete old backups by sending them to bin (and cleaning that one afterwards)

==Install Google Keep Notes as Application via Chromium==
First download [https://download-chromium.appspot.com/?platform=Mac&type=snapshots Chromium] and move to applications than follow this [http://www.rawinfopages.com/mac/content/how-run-google-keep-window-desktop-mac-app guide] to create a shell script for automator.
Type = Application
Action = Run Shell Script
Contents = /Applications/Chromium.app/Contents/MacOS/Chromium --app=https://keep.google.com
Save as .app in Application folder and drag to dock, done!

==Settings==
Hide top panel from secondary screen from [http://osxdaily.com/2013/10/27/hide-menu-bar-external-display-mac-os-x/]
Mission Control -> Uncheck the box next to 2Displays have separate Spaces"

==Tools==
===Git===
Git requires apple xcode tools
xcode-select --install

===HomeBrew Package Manager===
from [https://docs.brew.sh/FAQ]
First update the formulae and Homebrew itself:
brew update
install
brew install mypackage
list installed pakages
brew list
uninstall
brew remove mypackage
You can now find out what is outdated with:
brew outdated
Upgrade everything with:
brew upgrade
By default, Homebrew does not uninstall old versions of a formula, so over time you will accumulate old versions. To remove them, simply use:
brew cleanup mypackage
or clean up everything at once:
brew cleanup
or to see what would be cleaned up:
brew cleanup -n

====Special Commands====
show dependancies
brew deps gnuplot
search for a package/version
brew search gnuplot
manual download file into cache to install a local file [http://mygeekdaddy.net/2014/12/05/how-to-install-a-local-file-in-homebrew/]
ignore dependencies
brew install gnuplot --ignore-dependencies

===Eraze Disk/fill by Zeros===
diskutil zeroDisk /Volumes/myDiskName/
alternative: create a big file
dd if=/dev/zero of=tmp_file.txt

===Spelling: unlearn a word/edit custom words===
vim ~/Library/Spelling/LocalDictionary

==Apps==
* https://rectangleapp.com Window mover (Control+Command+Arrows)
* https://maccy.app clipboard manager (Command+Shift+C)
* https://apphousekitchen.com/ AlDente: charge limiter

Not in use
* https://github.com/exelban/stats/ system stats
* https://matthewpalmer.net/rocket/ emoji typing
* https://www.mowglii.com/itsycal/ tiny menu bar calendar
* https://macmousefix.com/ Mac Mouse Fix brings all features of an Apple Trackpad
* https://spoti.ca/ controls for Spotify and Apple Music

==Fixing problems==
this ssh warning
perl: warning: Setting locale failed.
can be fixed by setting this in .zshrc
export LC_ALL=en_US.UTF-8

==Terminal Hacks==
===Kill Process blocking a Port===
lsof -i :5173
kill 1234

Windows Batch Scripting

2026-04-17T17:07:23Z

Torben: /* Run as other user */

[[Category:Software]][[Category:Windows]]
[http://weblogs.asp.net/jongalloway/top-10-dos-batch-tips-yes-dos-batch see "Top 10 DOS Batch tips"] and [http://www.axel-hahn.de/batch/batchecke/tipps/ BATch-Dateien - kleine Tipps] for more stuff

===Template===
@echo off
title %~n0
setlocal

===current dir and filename===
# current working dir
echo %CD%
# path of current file
echo %0
# filename without the extension.
echo %~n0
# filename and extension.
echo %~nx0
# or
echo %~n0%~x0

===Title to Dirname===
@echo off
for %%f in (%cd%) do set dirname=%%~nxf
title %dirname%

===Empty c:\tmp===
@echo off
rmdir /q /s c:\tmp
# or rd /q /s c:\tmp
mkdir c:\tmp

===find files / search (and delete) old files===
via forfiles
forfiles /P "C:\tmp\test-del" /S /D -30 /C "cmd /c echo @path"
forfiles /P "C:\tmp\test-del" /S /D -30 /C "cmd /c del @path"
forfiles /P "C:\tmp\test-del" -S -D 90 -m *.log -c "cmd /c del @path"

via robocopy
robocopy "C:\tmp\test-del" "*.log" "C:\cleanup-logs" /mov /minage:90 /NP
del C:\cleanup-logs /q

===ReadOnly Flag / WriteProtection===
Remove
attrib -h -r -s /s /d *.*
Set
attrib +r c:\Users\torben\Desktop\SyncedFolder\*.* /s

R Read-only file attribute.
S System file attribute.
H Hidden file attribute.
/S Processes matching files in the current folder and all subfolders.
/D Processes folders as well.

===Set Current Working Dir to Script Location ===
This is important if the script is started using a taskmanager etc.
e:
cd %~dp0

===Set Window Title===
TITLE My Window Title
or when opening via START from another Batch file
START "My Window Title" 2.cmd

===Get FolderName===
This when excecuted in c:\sub\folder it returns "folder"
for %%* in (%CD%) do set CurrDirName=%%~nx*
echo %CurrDirName%

===List of files to Textfile===
dir *.* /b > ..\liste.txt

===Pipe output to textfile===
From [https://support.microsoft.com/en-us/help/110930/redirecting-error-messages-from-command-prompt-stderr-stdout] You can print the errors and standard output to a single file by using the "&1" command to redirect the output for STDERR to STDOUT and then sending the output from STDOUT to a file:
some_command > output.log 2>&1
some_command 1> output.log 2> error.log

===exit if command fails with error===
some_command1
IF %ERRORLEVEL% NEQ 0 exit /b %ERRORLEVEL%
some_command2
IF %ERRORLEVEL% NEQ 0 exit /b %ERRORLEVEL%

===Net use for network drive mounting===
net use Y: \\server\path /user:myuser

net use Y: /delete

===Zip */*.ini===
@echo off
setlocal EnableDelayedExpansion
set "OUT=inis.zip"
REM Use PowerShell to expand glob pattern and pipe relative paths to tar with --files-from=-
powershell -NoLogo -NoProfile -Command "Get-ChildItem -Path '*\*.ini' -File | ForEach-Object { $_.FullName.Substring((Get-Location).Path.Length + 1) } | tar -c -a -f '%OUT%' --files-from=-"

===Loops===
====if====
check if mounting (net use) was successful
net use q: \\server\share
q:
cd \
IF NOT EXIST dir1\dir2 (
echo mouting of share failed
pause
c:
net use q: /delete
EXIT /B 1
)

====for====
loop over values
for %%L in (cn, de, en, es, fr, ko, sk, vn) do (
echo %%L
)

loop over all dirs
FOR /D %%D in ("*") DO (
echo %%D
)
double loop over all dirs
FOR /D %%P in ("*") DO (
echo parent = %%P
FOR /D %%C in ("%%P\*") DO (
echo child = %%C
)
)

loop over files
for %%F in (*.pdf) do (
echo %%F
copy %%F %%~nF.old
)

====for and grep.exe====
using UnixUtils grep.exe one can easily extract exceptions from logs:
for %%F in (*.log) do (
grep -B3 -A3 "exception" %%F > %%~nF-error.txt
)

===Variables===
:: no spaces around '='!!!
set xyz=myfile.bat
set /p xyz=Variable Eingeben:
set /p xyz= < TMP.dat
echo %xyz%

====Substrings====
Substring via ":~"
set year=%date:~-4,4
Path, filename and extension of file stored in variable %%F
set path=%%~pF
set name=%%~nF
set ext=%%~cF

====Set Variable to Program Output====
for /f "delims=" %%a in (' powershell -c "$lastmonth = (Get-Date).addMonths(-3); $lastmonth.tostring(\"yyyy-MM\")" ') do set "MONTH=%%a"
echo %MONTH%

===Date & Time ===
====DateString====
set DATESTR=%date:~-4,4%-%date:~-7,2%-%date:~-10,2%_%time:~0,2%:%time:~3,2%:%time:~6,2%
or
set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%%time:~6,2%
:: replace ' ' in small hours with 0
set DATESTR=%DATESTR: =0%
= 240419_085724
zip.exe -9 %DATESTR%.zip *.bat
for better example of zipping see [[Backup#Zip_Folder|zip folder in Backup section]]

====DayOfWeek====
from [https://stackoverflow.com/questions/25537313/issue-with-getting-batch-script-to-print-keeps-given-a-error-about-set-was-unexp here]
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get dayofweek') DO (
if "!count!" GTR "0" GOTO next
set dow=%%D
SET /a count+=1
)
:next
echo %dow%

====Yesterday====
from [https://stackoverflow.com/questions/2954359/dos-batch-programming-howto-get-and-display-yesterday-date here]
set befehl="PowerShell $date = Get-Date; $date=$date.AddDays(-1); $date.ToString('yyyy-MM-dd')"
for /f %%i in ('%befehl%') do set yesterday=%%i

===Copy all files from subfolders to one folder ===
from [https://stackoverflow.com/questions/11720681/windows-batch-copy-files-from-subfolders-to-one-folder here]
set source=c:\source
set target=c:\target
cd %source%
for /r %%a in (*.*) do (
COPY "%%a" "%target%"
)

===FTP===
====FTP-Upload====
file "ftp.bat"
@echo off
ftp "-s:FtpScript"
pause
cls

file "FtpScript"
open www.xyz.de
[User]
[Password]
BINARY
put [File]
quit

===UnixUtils===
Using [https://sourceforge.net/projects/unxutils/ UnixUtils] for Windows wget, grep, etc are usable to scripts in Windows. In the UnixUtils.zip the .exe files are located in foltder usr/local/wbin/ . Examples:
head.exe -c 1000 mylog.log > outHead1000Bytes.log
head.exe -n 1000 mylog.log > outHead1000Lines.log
tail.exe -c 1000 mylog.log > outTail1000Bytes.log
tail.exe -n 1000 mylog.log > outTail1000Lines.log
grep.exe -i -B3 -A1 "ERROR" mylog.log > outGrepErrors.log

====Curl to check if page online====
@echo off
set URL1=https://my.server.com
:loop
for /f "delims=" %%i in ('curl -o nul -I -s -w "%%{http_code}" "%URL%"') do set HTTP_STATUS=%%i
echo Status: %HTTP_STATUS% at %date% %time%
timeout /t 5
goto loop

===TOP CPU RAM Disk I/O===
# CPU + RAM
top
# I/O
sudo iotop -o -d 2 -k

===Run as other user===
runas /user:MYDOMAIN\myuser /savecred "cmd /K cd c:\mydir && my_programm.exe"

/K to keep window open when finished
/C to close window when finished

===checks.cmd - Run chk_*.cmd===
run all chk_*.cmd scripts in same dir and print summary of failed ones

@echo off
title %~n0
setlocal enabledelayedexpansion
cd %~dp0\..

set failures=0
set "failed_list="

for %%f in (%~dp0chk_*.cmd) do (
echo.
echo === %%~nf ===
call "%%f"
if errorlevel 1 (
set /a failures+=1
set "failed_list=!failed_list! %%~nf"
)
)

echo.
if !failures!==0 (
echo All checks passed.
) else (
echo !failures! checks failed:!failed_list!
)

echo.
pause

Other scripts contain
...
some_command
if errorlevel 1 exit /b 1

Windows Batch Scripting

2026-04-17T17:04:49Z

Torben:

AI LLM for Coding

2026-04-16T10:24:03Z

Torben: /* Tips */

==Google Gemini==
pros: free to use with google account
create your API key at https://aistudio.google.com/app/api-keys

Install
npm install -g @google/gemini-cli@latest
gemini
# or just run
npx https://github.com/google-gemini/gemini-cli
usage
cd /tmp
export GEMINI_API_KEY="xxx"
git clone https://github.com/entorb/meeting-meter
cd myProject
gemini

to switch model from pro to flash:
export GEMINI_MODEL="gemini-2.5-flash"

secret alternative: .env file
GEMINI_API_KEY=xxx

==Google Antigravity==
https://antigravity.google/download

==Claude==
===Web===
https://claude.ai 
pro: Daily reset free quota 
cons: need to manually upload single files

===Console===
Claude Console via API and VISA Pre-Paid: https://console.anthropic.com 
pro: fully integrated in code base 
cons: costs money, 5€ are easily spend
npm install -g @anthropic-ai/claude-code
cd myProject
claude

==Tips==
* E2E tests (e.g. Cypress) are very important
* Use tools for code formatting, linting, type-hints
* Configure these tools to produce minimum output to reduce token consumption. e.g. for
** Prettier: --log-level silent
** Cypress: cypress run --e2e --quiet
* For Refactoring/renaming: better do it manually, via vscode, that is a lot faster

===Cavemen style===
https://github.com/JuliusBrussee/caveman
npx skills add JuliusBrussee/caveman -a github-copilot
It has an option to install to local home dir instead of to project repo.

Python

2026-04-13T12:48:47Z

Torben: /* Vulture: Search for dead code */

[[Category:Coding]][[Category:Python]]

==Getting Started==
===Install===
====Python====
Best use [https://docs.astral.sh/uv/ UV] for managing python and package versions.

* for Windows: get and install Python from https://www.python.org
* for MacOS: use [https://github.com/pyenv/pyenv?tab=readme-ov-file#a-getting-pyenv pyenv]
brew install xz
brew install pyenv

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[ [ -d $PYENV_ROOT/bin ] ] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init - zsh)"' >> ~/.zshrc
exec "$SHELL"

pyenv install --list
pyenv install 3.13.5
pyenv global 3.13.5

vim ~/.zshrc
# add
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi

====Editor: Visual Studio Code====
excellent and free source-code editor that supports many languages.
* get and install from https://code.visualstudio.com

See Wickie page [[Visual_Studio_Code]] for general setup, extension, config...

Python Extensions
* Python
* Pylance
* [https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff Ruff Formater and Linter]
* [https://marketplace.visualstudio.com/items/?itemName=tamasfe.even-better-toml Even Better TOML]
Settings (CTRL + ,)
* Extensions -> Python -> Formatting: Provider = black
* Text Editor -> Editor: Format On Save
* Text Editor -> Files:Eol -> \n
* Linter: Ruff
Settings in settings.json
"python.analysis.completeFunctionParens": true,
"python.analysis.autoImportCompletions": true,
"python.analysis.inlayHints.functionReturnTypes": true,
"python.analysis.typeCheckingMode": "strict",
// Ruff
"[python]": {
"editor.defaultFormatter": "charliermarsh.ruff",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.fixAll": "explicit",
"source.organizeImports.ruff": "explicit"
},
},

Run your code
CTRL + F5 : run
F5 : run in debugger

====Code Formatter Ruff====
use software for handling the code formatting like "ruff" or "black", where Ruff is much faster and additionally provides linting.
pip install ruff
than activate in editor like vs code

===My Template===
see latest version at https://github.com/entorb/template-python

"""Main file."""
# by Torben Menke https://entorb.net
import logging
from pathlib import Path

def init_logging() -> None:
"""Initialize logging."""
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(
level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s"
)
logging.getLogger("httpx").setLevel(logging.WARNING)

init_logging()
LOGGER = logging.getLogger(__name__)

def main():
LOGGER.info("Moin Moin")

if __name__ == "__main__":
main()

"""Other file"""
import logging
LOGGER = logging.getLogger(__name__)

===Compile to .exe===
pip install pyinstaller
pyinstaller --onefile --console your.py
# for Excel and Matplotlib these options are required
--hidden-import=openpyxl --hidden-import=matplotlib --hidden-import pandas.plotting._matplotlib
([[Python - py2exe]] is deprecated)

==Basics==
=== Naming Conventions===
[https://google.github.io/styleguide/pyguide.html Google Python Style Guide]:
module_name, package_name, ClassName, method_name, ExceptionName, function_name, GLOBAL_CONSTANT_NAME, global_var_name, instance_var_name, function_parameter_name, local_var_name

====Doc Strings====
It is best practice to start a file with a docstring:
"""My Script"""
This can be accessed linke this:
title = __doc__[:-1] # type: ignore

===Installing packages===
python -m pip install --upgrade pip

pip install somemodule
# or
pip3 install somemodule
# or read from file
pip install -r requirements.txt

# uninstall
pip uninstall somemodule

# using a web proxy
# set proxy for windows cmd session
SET HTTPS_PROXY=http://myProxy:8080
(afterwards --proxy setting below no longer required
or
pip install --proxy http://myProxy:8080 somemodule

# list outdated packages
pip list --outdated

# update package
pip install --upgrade pyinstaller

# updating all via Windows Powershell (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | %{$_.split('==')[0]} | %{pip install --upgrade $_}

# updating all via Bash (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install --upgrade

# downgrade
pip install --upgrade pandas==1.2.4

====Oneline Updater for requirements.txt====
https://pkgui.com/pip

====update packages====
pip install pip-review
pip-review --auto

====generate requirements.txt====
pip install pipreqs
pipreqs ./

===Loops===
ATTENTION: The loops do not create a new variable scope. Only functions and modules introduce a new scope!
for p in all_files:
print(p)

for i, p in enumerate(all_files):
print(i, p)

for i in range(10):
print(i)
# del i

while i <= 100:
i += 1
...
if sth1:
continue # start next loop
if sth2:
break # exit loop

# inline if (requires a dummy else):
print("something") if sth else 0

===Math===
see [[Python - Math]] for linear regression

Modulo
15 % 4
# > 3

Integer Division
17 // 4
# > 4

====Random====
import random
random.randint(1000000, 9999999)

==Basic Objects==
===Variables===
del var # delete / undef a variable
var = None # sets to null

# check if variable is defined
if "var" in locals():
pass
# for object oriented projects:
if "var" in self.__dict__:
pass

# Access global variables in functions
var = 123

def test() -> None:
global var # point to global instead of creation of local var
var = 321

===Strings===
# num <-> str
s = str(i) # int to string
f = float(s) # str -> float
i = int(s)
str(round(f, 1)) # round first
# tests
s.isdigit() # 0-9
# note isdecimal() does also not match '1.1'

# printf: 1 digit
s = f"{value:0.1f}"
s = "%0.1f" % value

====Modify Strings====
# get string from prompt
s = input("Enter Text: ")

s = s.strip() # trim spaces from both sides, rstrip for right only
s = s.lower() # lower case
s = s.upper() # upper case
s = s.title() # upper case for first char of word

# upper case first letter of each word and also removes multiple and trailing spaces
import string
s = string.capwords(s)

# replace
s.replace(x, y)

# trim whitespaces from left and right
s.strip()

# replace all (multiple) whitespaces by single space ' '
s = " ".join(s.split())

# generate key value pairs from dict
# key1=value1&key2=value2
param_str = "&".join("=".join(tup) for tup in dict.items())

# repeat string multiple times
s * 5 # = s+s+s+s+s

====Substrings====
# find a substring:
if x in s: # True / False
pass
if len(s) > 0:
pass

# handling substrings
a = "abcd"
b = a[:1] + "o" + a[2:]
# > 'aocd'

s = "Hello there !bob@"
i1 = s.find("!") + 1
i2 = s.find("@")
substr = s[i1:i2]

def substr_between(s: str, s1: str, s2: str) -> str:
assert s1 in s, f"E: can't find '{s1}' in '{s}'"
assert s2 in s, f"E: can't find '{s1}' in '{s}'"
i1 = s.find(s1) + len(s1)
i2 = s.find(s2)
assert i1 < i2, f"E: '{s1}' not before '{s2}' in '{s}'"
return s[i1:i2]

====Binary, raw strings, html encoding====
# Binary Strings
sb = b"asdf"
# or
sb = str.encode("asdf")
s = sb.decode("ascii") # decode binary strings
sb = s.encode("ascii") # encode string to binary

# raw string
s = r"c:\Windows" # no escape of \ needed

# convert utf-8 to html umlaute
s = "Nürnberg".encode("ascii", "xmlcharrefreplace").decode()
# -> Nürnberg

====Guess encoding via chardet====
import chardet # pip install chardet
result = chardet.detect(raw_data)
if result["encoding"] and result["confidence"] > 0.5: # noqa: PLR2004
encoding = result["encoding"]
else:
encoding = "utf-8"

there is a much faster alternative [https://github.com/PyYoshi/cChardet cchardet], but that requires Microsoft Visual C++ 14.0

====Merge variables in string / sprintf====
print("Renner =", i)
print("Renner = %3d" % i) # leading 0's
print(f"Renner = {i}")

# place formatted numbers in a string / sprintf
s = "The {:.1f}% {} cost {:05.2f} euros".format( 5.1, "beer", 3.50)
print(s)
# > The 5.1% beer cost 03.50 euros

s = f"The length is {72.8958:.2f} meters"
# > The length is 72.90 meters

===Lists===
like @array in Perl
lst = [1, 2, 3, 4, 5, 6]
lst = [x / 2 for x in range(10)]
lst = [None] * 10 # Initiate list of None elements:
len(lst) # length
lst2 = lst[0:10] # get elements 0-10
for i in lst:
print(i)

====clone list====
# dangerous: creates a link to the original list
list_2 = my_list # M's elements are LINKS to L's
# clones can be achieved via:
list_2 = my_list.copy
list_2 = my_list[:]
list_2 = list(my_list)

====list modifications====
lst.pop() # returns and removes the last item
lst.pop(i) # returns and removes the item at position int i
lst.insert(i, x) # insert item x at position int i
lst.remove(x) # removes the first occurrence of item x
lst.append(x) # append a single element
lst.extend(m) # append elements of another list

lst.reverse() # reverse the order of the list
lst = sorted(["B", "a", "c"], key=str.casefold) # case insensitive / ignore case

# list to string
s = "".join(lst)
s = "\n".join(lst)
# string to list
lst = s.split() # default: split on space
lst = s.split(",")

# remove empty values from end
while L[-1] == "":
L.pop()

# check and remove items from list
# from https://stackoverflow.com/a/6024599
# iterates in-situ via reversed index
for i in range(len(lst) - 1, -1, -1):
element = lst[i]
if check(element):
del lst[i]

====check if item is in list====
x in lst
x not in lst
lst.count(x) # how many items x are in the list
i = lst.index("word") # find in list / returns the position of the first match in list

====pair 2 lists====
# zip: merge 2 lists to list of tuples
data = list(zip(data_x, data_y, strict=True))

# unzip: split list of pairs into 2 lists
data_x, data_y = zip(*data, strict=True)

# Cartesian product of lists / tuples
import itertools
for i in itertools.product(*list_of_lists):
print(i)

====sort multidim list====
lst = sorted(lst, key=lambda x: x[0], reverse=False)
lst = sorted(lst, key=lambda row: (row["Wann"], row ["Wer"]), reverse=False)

====filter list====
lines = [x for x in lines if not x.startswith ("word")]
lst = ["asdf", "asdf2", "qwertz"]
lst = [elem for elem in lst if "asdf" in elem]

====for each element====
# trim/strip spaces for each element
lines = [s.strip() for s in lines]
# modify each item in list by adding constant string
l = [s + ';' + v for v in l]

# modify item in list
for idx, line in enumerate(cont):
if "K1001/1" in line:
line = "K1001/1 Test Nr " + str(i) + "\n"
cont[idx] = line
break

===Tuples===
Ordered sequence, with no ability to replace or delete items
tpl = (1,2,3,4,5,6)

list -> tuple
tpl = tuple(lst)

combine 2 tuples
tpl = tpl1 + tpl2

===Dictionaries===
like %hash in Perl
d = {"key1": "value1", "key2": 123}
d["key3"] = (1, 2, 3)
del d["key3"]

len(d)
d.clear()
d.copy()
d.keys()
d.values()
d.items() # returns a list of tuples (key, value)
d.get(k) # returns value of key k
d.get(k, x) # returns value of key k; if k is not in d it returns x
count = d.get("key", 0) + 1 # nice for counters
d.pop(k) # returns and removes item k
d.pop(k, x) # returns and removes item k; if k is not in d it returns x
# checks
x in d
x not in d

# dict can have tuple as key
tpl = ("key1", "key2", 123)
d[tpl] = 123456

# loop over all key-value pairs
for key, value in d.items():
print(f"{key} = {value}")

# sorted keys:
for key in sorted(dict.keys()):
pass
# sorted values, reversed
for key, value in sorted(d.items(), key=lambda item: item[1], reverse=True):
pass

# join / merge 2 dicts
d.update(d2)

# flatten dict of dict
flat = d.copy()
meta = d.pop("metadata")
flat.update(meta)

====MultiDim Dictionaries====
d = {}
d["item1"] = {}
d["item1"]["value"] = 1.909e18
d["item1"]["name"] = "Item 1"
d["item2"] = {}
d["item2"]["value"] = 1.725e18
d["item2"]["name"] = "Item 2"

for k in d.keys():
d[k]["value"] = d[k]["value"] * 1.1

def find_common_strings(dict1: dict, dict2: dict) -> dict[Any, Any]:
"""Find common strings in two dict, returning a new dict of those strings."""

def recursive_match(d1: dict, d2: dict) -> dict[Any, Any]:
common_dict = {}
for key in d1: # noqa: PLC0206
if key in d2:
if isinstance(d1[key], dict) and isinstance(d2[key], dict):
# Recurse if both values are dicts
nested_common = recursive_match(d1[key], d2[key])
if nested_common: # Add to result if common values are found
common_dict[key] = nested_common
elif isinstance(d1[key], str) and isinstance(d2[key], str):
# Check if both are strings and identical
if d1[key] == d2[key]:
common_dict[key] = d1[key]
return common_dict

return recursive_match(dict1, dict2)

def update_dict_with_new_values(original_dict: dict, new_values: dict) -> dict:
"""Update strings in the original dict with new values from the new_values dict."""

def recursive_update(d1: dict, new_vals: dict) -> None:
for key in new_vals: # noqa: PLC0206
if key in d1:
if isinstance(d1[key], dict) and isinstance(new_vals[key], dict):
# Recurse if both values are dicts
recursive_update(d1[key], new_vals[key])
elif isinstance(d1[key], str) and isinstance(new_vals[key], str):
# Update the string
d1[key] = new_vals[key]

recursive_update(original_dict, new_values)
return original_dict

===Datetime, Date and Time===
import datetime as dt
from zoneinfo import ZoneInfo
TZ_DE = ZoneInfo("Europe/Berlin")
TZ_UTC = ZoneInfo("UTC")

====Create====
# date
date = dt.date(2023, 12, 31)
date = dt.date.fromisocalendar(int(year), int(week), int(daynum)) # daynum: 1..7
date = dt.date.fromisoformat("2020-03-10")
date = dt.datetime.strptime(datestr, "%y%m%d").date()
date_today = dt.datetime.now(tz=dt.UTC).date()
date_yesterday = dt.datetime.now(tz=TZ_DE).date() - dt.timedelta(days=1)
days_overdue = (date_completed - date_due).days
# to add one month (which is not supported by timedelta)
from dateutil.relativedelta import relativedelta
date_end = date_start + relativedelta(months=1)
delta_days = 1 + (end - start).days

# datetime
dt_now = dt.datetime.now(tz=TZ_DE)
my_dt = dt.datetime(2023, 12, 31, 14, 31, 56, tzinfo=TZ_DE) # 2023-12-31 14:31:56
my_dt = dt.datetime.fromtimestamp(my_timestamp, tz=TZ_DE)
my_dt = dt.datetime.fromisoformat ("2017-01-01T12:30:59.000000")
my_dt = dt.datetime.fromisoformat("2020-03-10 06:01:01+00:00")
s = "2020-03-10T06:01:01Z"
my_dt = dt.datetime.fromisoformat(s.replace("Z", " +00:00"))
my_date_local = my_dt.astimezone(tz=TZ_DE).date()

# datetime -> date
my_date = my_dt.date()
# date -> datetime
my_date = dt.date(2023, 12, 31)
my_dt = dt.datetime(my_date.year, my_date.month, my_date.day, tzinfo=TZ_DE)

# to string
date_str = my_dt.strftime("%y%m%d")
date_str = my_dt.strftime("%Y-%m-%d %H:%M:%S")
# alternative using time
date_str = time.strftime("%Y-%m-%d %H:%M:%S")
# now in UTC without milliseconds
date_str = dt.datetime.now(tz=dt.timezone.utc).replace(microsecond=0).isoformat() + "Z"

# remove timezone
my_dt = my_dt.replace(tzinfo=None)

# German format
import locale
locale.setlocale(locale.LC_ALL, "de_DE")
print(my_date.strftime("%a %x")) # Mo 25.12.2023

# Calendar week
week = my_date.isocalendar()[1]
print("KW%02d" % week)

# first day of quarter
def get_first_day_of_the_quarter(my_date: dt.date) -> dt.date:
return dt.date(my_date.year, 1 + 3 * ((my_date.month - 1) // 3), 1)

====rounding datetimes to minutes====
def floor_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Floor (=round down) minutes to X min resolution."""
min_new = res * (my_dt.minute // res)
return my_dt.replace(minute=min_new, second=0, microsecond=0)

def ceil_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Ceil (=round up) minutes to X min resolution."""
min_new = res * (1 + my_dt.minute // res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

def round_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Cound minutes to X min resolution."""
min_old_dec = float(my_dt.minute) + float(my_dt.second * 60)
min_new = res * round(min_old_dec / res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

my_dt = dt.datetime.fromisoformat("2011-11-04 00:05:23+00:00")
print(f"original: {my_dt}")
print(f"floored: {floor_dt_minutes(my_dt,5)}")
print(f"ceileded: {ceil_dt_minutes(my_dt,5)}")
print(f"rounded: {round_dt_minutes(my_dt,5)}")

====Timing / Time elapsed====
import time
timestart = time.time()
sec = time.time() - timestart
print(f"Time elapsed: {sec:.2f} sec")

==OS, Argpars, etc.==
===Hostname===
from socket import gethostname
print(gethostname())

===Checking Operating System===
import os
import sys

if os.name == "posix":
print("posix/Unix/Linux")
elif os.name == "nt":
print("windows")
else:
print("unknown os")
sys.exit(1) # throws exception, use quit() to close / die silently

===Get filename of python script===
my_file_path = __file__

alternative using sys package
from sys import argv
myFilename = argv[0]

===accessing os envrionment variables===
import os
print(os.getenv("tmp"))
# better:
try:
s = os.environ[key] # throws error if unset
except KeyError:
error_message(f"Environment variable {key} not set.")

===Command Line Arguments===
====ArgumentParser====
import argparse

parser = (
argparse.ArgumentParser()
) # construct the argument parser and parse the arguments
# -h comes automatically

# Boolean Parameter
parser.add_argument(
"-v", "--verbose", help="increase output verbosity", action="store_true"
) # store_true -> Boolean Value

# Choice Parameter
# restrict to a list of possible values / choices
# parser.add_argument("--choice", type=int, choices= [0, 1, 2], help="Test choices")

# Positional Parameter (like text.py 123)
# parser.add_argument("num", type=int, help="Number of things")

# Required Parameter
# parser.add_argument("-i", "--input", type=str, required=True, help="Path of file")

# Optional Parameter
parser.add_argument("-n", "--number", type=int, help="Number of clicks")
# Optional Parameter with Default
parser.add_argument(
"-s",
"--seconds",
type=int,
default=sec_default,
help="Duration of clicking, default = %i (sec)" % sec_default,
)

args = vars(parser.parse_args())

if args["verbose"]:
pass # do nothing
# print ("verbosity turned on")
if args["number"]:
print("num=%i" % args["number"])

==== match case statement ====
(new in python 3.10)
match args:
case {"sap": "prod", "version": 1}:
...
case {"sap": "prod", "version": 2}:
...
case _: # default case

==File Access==
===Pathlib===
for migrating see table [https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module]
from pathlib import Path

p = Path("dir/test.txt")

log_file = Path(__file__).with_suffix(".log") # __file__ is name of python script

my_fname = p.name # alternative to basename
my_ext = p.suffix
(filepath_without_ext, file_ext) = (p.stem, p.suffix)
my_dir = p.parent
my_parent_dir = p.parents[1]
p2 = p.with_suffix(".json")
p3 = p.parent / (p.name + "-autofix.tex")

# checks
Path(my_file_str).exists()
Path(my_file_str).is_file()
Path(my_file_str).is_dir()

# loop over glob of files matching wildcard
for file_out in Path("mydir").glob("*-autofix.tex"):

# loop over dirs
p = Path() # cwd
list_of_repos = [x for x in p.iterdir() if x.is_dir() and (x / ".git").is_dir()]

alternative using old os functions
Split path into folder, filename, ext
import os
(dirName, fileName) = os.path.split(f)
(fileBaseName, fileExtension) = os.path.splitext(fileName)
fileOut = os.path.splitext(fileIn)[0] + "-out.txt"

===File Manipulations: copy, move, delete, touch===
from pathlib import Path

# copy
from shutil import copyfile
copyfile(Path("file-source.txt"), Path("dir") / Path("file-target.txt"))

# move / rename
Path("file.txt").replace(Path("file2.txt"))

# delete
Path("file.txt").unlink()

# touch
Path("file.txt").touch()

===File size and timestamp===
# size
Path("file.txt").stat().st_size

# timestamp last modified
Path("file.txt").stat().st_mtime

===Dir create/mkdir and delete===
# create dir
from pathlib import Path
Path("myDir1/myDir2").mkdir(parents=True, exist_ok=True)

# delete dir
import shutil
shutil.rmtree(d)

===Loop over Directories===
====Fetch Dir Contents / Loop over Files====
glob via pathlib
for fileOut in Path("mydir").glob("*-autofix.tex"):

Get list of files in directory, filter dirs from list, filter by ext
dir= "/path/to/some/dir"
listoffiles = [ f for f in os.listdir(dir) if os.path.isfile(os.path.join(dir ,f)) and f.lower()[-4:] == ".gpx"]
listoffiles.sort()

====Traverse in Subdirs====
# walk into path an fetch all files matching extension jpe?g
files = []
for (dirpath, dirnames, filenames) in os.walk("."):
dirpath = dirpath.replace("\\", "/")
for file in filenames:
if file.endswith(".txt"):
files.append(dirpath + "/" + file)
elif re.search(r"\.jpe?g$", file, re.IGNORECASE):
files.append(dirpath + "/" + file)

==File Parsing==
===File General===
====File Read====
path_file_in = Path("file.txt")

# via pathlib
cont = path_file_in.read_text(encoding="utf-8") # note: utf-8-sig for UTF-8-BOM

# general
with path_file_in.open(encoding="utf-8") as fh:
cont = fh.read()
# or
lst = fh.readlines()
# or
line = fh.readline()
# or
for line in fh:
print(line)

fh = path_file_in.open(encoding="utf-8")
...
fh.close()

====File Write====
path_file_out = Path("out/1/out.txt")

# write via pathlib
path_file_out.write_text("some text")

# write general
with path_file_out.open(mode="w", encoding="utf-8", newline="\n") as fh:
# w = overWrite file ; a = append to file
# If running Python in Windows, "\n" is automatically replaced by "\r\n".
# To prevent this use newline='\n'
fh.writelines(lst) # no linebreaks
# or
fh.write("\n".join(lst))
# or
for line in lst:
fh.write(line)

# Force update of file contents without closing it
fh.flush()

# alternative
fh = path_file_out.open(mode="w", encoding="utf-8", newline="\n")
...
fh.close()

===CSV===
====CSV Read====
import csv
from pathlib import Path

with Path("data.csv").open(mode="r", encoding="utf-8") as fh: # for Excel use ANSI
csv_reader = csv.DictReader(fh, dialect="excel", delimiter="\t")
for row in csv_reader:
print(f'\t{row["name"]} works in the {row["department"]} department')

====CSV Write====
import csv
from pathlib import Path

# simple
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.writer(fh, delimiter="\t")
csvwriter.writerow(("Date", "Confirmed"))

# dictwriter
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.DictWriter(
fh,
delimiter="\t",
extrasaction="ignore",
fieldnames=["date", "occupied_percent", "occupied", "total"],
)
csvwriter.writeheader()
for d in my_list_of_dicts:
d["occupied_percent"] = round(100 * d["occupied"] / d["total"], 1)
csvwriter.writerow(d)

===JSON===
====JSON Read====
import json
# from file
with path_to_download_file.open(encoding="utf-8") as fh:
d_json = json.load(fh)
# from string
d_json = json.loads(response_text)

def json_read(file_path: Path) -> list[dict[str, str]]:
"""
Read JSON data from file.
"""
with file_path.open(encoding="utf-8") as fh:
json_data = json.load(fh)
return json_data

====JSON Write====
Write dict to file in JSON format, keeping utf-8 encoding
import json

with path_to_download_file.open(mode="w", encoding="utf-8", newline="\n") as fh:
json.dump(my_dict, fh, ensure_ascii=False, sort_keys=False, indent=2)

def json_write(file_path: Path, json_data: list[dict[str, str]]) -> None:
"""
Write JSON data to file.
"""
with file_path.open("w", encoding="utf-8", newline="\n") as fh:
json.dump(json_data, fh, ensure_ascii=False, sort_keys=False, indent=2)

===Excel===
====Excel Read====
import openpyxl
# or xlsxwriter for write-only

workbook = openpyxl.load_workbook(
pathToMyExcelFile,
data_only=True, # read values instead of formulas
read_only=True, # suppresses: "UserWarning: wmf image format is not supported so the image is being dropped"
)
sheet = workbook["mySheetName"]
# or fetch active sheet
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=34, column=1) # index start here with 1
print(cell.value)
# or
print(sheet.cell(column=col, row=row).value)

====Excel Write====
import openpyxl

workbook = openpyxl.Workbook()
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=i, column=j) # index starts at 1
cell.value = "asdf"
workbook.save("out.xlsx")

===Word===
====Word Read====
from io import BytesIO
from pathlib import Path
import pandas as pd
from docx import Document
from docx.document import Document as DocxDocument

def read_doc_to_df(p_doc: Path | BytesIO) -> pd.DataFrame: # noqa: C901, PLR0912
"""Read Word document to DataFrame."""
if isinstance(p_doc, Path):
doc = Document(str(p_doc))
elif isinstance(p_doc, BytesIO):
doc = Document(p_doc)

# list of multiple paragraphs per key
data: dict[str, dict[str, list[str]]] = {}

sections1: list[str] = []
section1 = ""
key = ""
section2 = ""
score = ""

for paragraph in doc.paragraphs:
assert paragraph.style is not None
text = paragraph.text.strip()
if paragraph.style.name == "Title": # doc title
continue
if paragraph.style.name == "Heading 1": # sheet name
section1 = text
if section1 not in sections1:
sections1.append(section1)
key = ""
section2 = ""
elif paragraph.style.name == "Heading 2": # Key of the question
key = text.split(" | ")[0]
key = section1 + "|" + key
section2 = ""
elif paragraph.style.name == "Normal" and text.startswith("[") and "]" in text:
section2, t2 = text[1:].split("]", maxsplit=1)
if section2 == "Score":
score = t2.strip()
data[key][section2] = [score]
elif paragraph.style.name == "Normal":
text = paragraph.text.strip()
# skip requirements leftovers and empty paragraphs
if (section2 == "" and text.startswith("...")) or text == "":
continue
lst = data.setdefault(key, {}).setdefault(section2, [])
lst.append(text)
data[key][section2] = lst
else:
msg = f"Unexpected style: {paragraph.style.name} in paragraph: {text}"
raise ValueError(msg)

# flatten the text from list to str
data2: dict[str, dict[str, str]] = {}
for key, sections in data.items():
for section2, lst in sections.items():
s = "\n".join(lst)
if key not in data2:
data2[key] = {}
data2[key][section2] = s.strip()

df1 = pd.DataFrame.from_dict(data2, orient="index")
df1.index.name = "Key"
df1 = df1.reset_index()
df1[ [ "Sheet", "Key" ] ] = df1["Key"].str.split("|", n=1, expand=True)

return df1

====Word Write====
from io import BytesIO
from pathlib import Path
import pandas as pd from docx import Document
from docx.document import Document as DocxDocument
from docx.shared import Cm

LAYOUT_SMALL_MARGINS = True
if LAYOUT_SMALL_MARGINS:
sections = doc.sections
for section in sections:
section.top_margin = Cm(1)
section.bottom_margin = Cm(1)
section.left_margin = Cm(1)
section.right_margin = Cm(1)

doc.add_heading("My Title", level=0)
doc.add_heading("My Section", level=1)
doc.add_paragraph("Some Text)
p = doc.add_paragraph()
runner = p.add_run(bold text")
runner.bold = True # runner.italic = True

==Regular Expressions==
See [http://docs.python.org/library/re.html]

See [https://pythex.org] for an online tester

multiple flags are joined via pipe |
s = re.sub("asdf.*", r"qwertz", s, flags=re.DOTALL | re.IGNORECASE)

====Lookahead and Lookbehind====
pos lookahead: (?=...)
neg lookahead: (?!...)
pos lookbehind (?<=...)
neg lookbehind (?<!...)

====matching====
import re

# V0: simple 1
myPattern = "(/\*\*\* 0097_210000_0192539580000_2898977_0050 \*\*\*/.*?)($|/\*\*\*)"
myRegExp = re.compile(myPattern, re.DOTALL)
myMatch = myRegExp.search(cont)
assert myMatch != None, f"golden file not found in file {filename}"
cont_golden = myMatch.group(1)

# V1: simple 2
assert (
re.match("^[a-z]{2}$", d_settings["country"]) != None
), f'Error: county must be 2 digit lower case. We got: {d_settings["country"]}'

# V2: find and count
was = r"""Part: ([^<+])"""
cnt_parts = len(re.findall(was, cont))
cont = re.sub(was, r"\n\nTeil: \1\n\n", cont)
assert cnt_parts == 4, f"{cnt_parts} == 4"
assert "Part:" not in cont

=====Match email=====
def checkValidEMail(email: str) -> bool:
# from https://stackoverflow.com/posts/719543/timeline bottom edit
if not re.fullmatch(r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$", email):
print("Error: invalid email")
quit()
return True

====Find all====
myMatches = re.findall('href="([^"]+)"', cont)
for myMatch in myMatches:
print(myMatch)

====substring====
import re

# simple via search
lk_id = re.search('^.*timeseries\-(\d+)\.json$', f).group(1)

# simple via sub
myPattern = "^.*" + s1 + "(.*)" + s2 + ".*$"
out = re.sub(myPattern, r"\1", s)

# more robust including an assert
def substr_between(s: str, s1: str, s2: str) -> str:
"""
returns substring of s, found between strings s1 and s2
s1 and s2 can be regular expressions
"""
myPattern = s1 + '(.*)' + s2
myRegExp = re.compile(myPattern)
myMatches = myRegExp.search(s)
assert myMatches != None, f"E: can't find '{s1}'...'{s2}' in '{s}'"
out = myMatches.group(1)
return out

matchObj = re.search(r"(\d+\.\d+)", text
if matchObj:
price = float( '%s' % (matchObj).group(0) )

====Naming of match groups====
(?P<name>...), see [https://docs.python.org/3/library/re.html]

====Search and Replace====
From [http://www.regular-expressions.info/python.html] 
re.sub(regex, replacement, str) performs a search-and-replace across subject, replacing all matches of regex in str with replacement. The result is returned by the sub() function. The str string you pass is not modified.

s = re.sub(" +", " ", s)

====Splitting====
From [http://docs.python.org/library/re.html] 
split() splits a string into a list delimited by the passed pattern. The method is invaluable for converting textual data into data structures that can be easily read and modified by Python as demonstrated in the following example that creates a phonebook.

First, here is the input. Normally it may come from a file, here we are using triple-quoted string syntax:
>>> input = """Ross McFluff: 834.345.1254 155 Elm Street
...
... Ronald Heathmore: 892.345.3428 436 Finley Avenue
... Frank Burger: 925.541.7625 662 South Dogwood Way
...
...
... Heather Albrecht: 548.326.4584 919 Park Place"""

The entries are separated by one or more newlines. Now we convert the string into a list with each nonempty line having its own entry:
>>> entries = re.split("\n+", input)
>>> entries
['Ross McFluff: 834.345.1254 155 Elm Street',
'Ronald Heathmore: 892.345.3428 436 Finley Avenue',
'Frank Burger: 925.541.7625 662 South Dogwood Way',
'Heather Albrecht: 548.326.4584 919 Park Place']

Finally, split each entry into a list with first name, last name, telephone number, and address. We use the maxsplit parameter of split() because the address has spaces, our splitting pattern, in it:
>>> [re.split(":? ", entry, 3) for entry in entries]
[['Ross', 'McFluff', '834.345.1254', '155 Elm Street'],
['Ronald', 'Heathmore', '892.345.3428', '436 Finley Avenue'],
['Frank', 'Burger', '925.541.7625', '662 South Dogwood Way'],
['Heather', 'Albrecht', '548.326.4584', '919 Park Place']]

==== replace cont by linebreaks====
def replace_cont_by_linebreaks(s: str, regex: str) -> str:
"""
Replace regex in s by the number of linebreaks it originally contained.
"""
myMatches = re.findall(regex, s, flags=re.DOTALL)
for match in myMatches:
linebreaks = match.count("\n")
s = s.replace(match, "\n" * linebreaks, 1)
return s

==Image/Picture/Photo==
from PIL import Image, ImageFilter # pip install Pillow

fileIn = "2018-02-09 13.56.25.jpg"
# Read image
img = Image.open(fileIn)
PROBLEM: 
PIL Image.save() drops the IPTC data like tags, keywords, copywrite, ... 
better using https://imagemagick.org instead when tags shall be kept

====Resize====
# Resize keeping aspect ration -> img.thumbnail
# drops exif data, exif can be added from source file via exif= in save, see below
size = 1920, 1920
img.thumbnail(size, Image.ANTIALIAS)

====Export file====
fileOut = os.path.splitext(fileIn)[0] + "-edit.jpg"
try:
img = Image.open(fileIn)
img.save(fp=fileOut, format="JPEG", quality='keep') # exif=dict_exif_bytes
# JPEG Parameters
# * qualitiy : 'keep' or 1 (worst) to 95 (best), default = 75. Values above 95 should be avoided.
# * dpi : tuple of integers representing the pixel density, (x,y)
except IOError:
print("cannot write file '%s'" % fileOut)

====Export Progressive / web optimized JPEG====
from PIL import ImageFile # for MAXBLOCK for progressive export
fileOut = os.path.splitext(fileIn)[0] + "-progressive.jpg"
try:
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)
except IOError:
ImageFile.MAXBLOCK = img.size[0] * img.size[1]
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)

====JPEG Meta Data: EXIF and IPTC====
=====IPTC: Tags/Keywords=====
from iptcinfo3 import IPTCInfo # this works in pyhton 3!
iptc = IPTCInfo(fileIn)
if len(iptc['keywords']) > 0: # or supplementalCategories or contacts
print('====> Keywords')
for key in sorted(iptc['keywords']):
s = key.decode('ascii') # decode binary strings
print(s)

=====EXIF via piexif=====
import piexif # pip install piexif
exif_dict = piexif.load(img.info['exif'])
print(exif_dict['GPS'][piexif.GPSIFD.GPSAltitude])
# returns list of 2 integers: value and donator -> v / d
# (340000, 1000) => 340m
# (51, 2) => 25.5m

# Modify altitude
exif_dict['GPS'][piexif.GPSIFD.GPSAltitude] = (140, 1) # 140m

# write to file
exif_bytes = piexif.dump(exif_dict)
fileOut = os.path.splitext(fileIn)[0] + "-modExif.jpg"
try:
img.save(fp=fileOut, format="jpeg", exif=exif_bytes, quality='keep')
except IOError:
print("cannot write file '%s'" % fileOut)
or
exif_dict = piexif.load(fileIn)
for ifd in ("0th", "Exif", "GPS", "1st"):
print("===" + ifd)
for tag in exif_dict[ifd]:
print(piexif.TAGS[ifd][tag]["name"], "\t",
tag, "\t", exif_dict[ifd][tag])
print(exif_dict['0th'][306]) # 306 = DateTime

=====EXIF via exifread=====
# Open image file for reading (binary mode)
fh = open(fileIn, "rb")
# Return Exif tags
exif = exifread.process_file(fh)
fh.close()
# for tag in exif.keys():
# if tag not in ('JPEGThumbnail', 'TIFFThumbnail', 'Filename', 'EXIF MakerNote'):
# print("%s\t%s" % (tag, exif[tag]))
print(exif["Image DateTime"])
print(exif["GPS GPSLatitude"])
print(exif["GPS GPSLongitude"])

=====EXIF GPS via PIL=====
# from https://developer.here.com/blog/getting-started-with-geocoding-exif-image-metadata-in-python3
def get_exif(filename):
image = Image.open(filename)
image.verify()
image.close()
return image._getexif()

def get_labeled_exif(exif):
labeled = {}
for (key, val) in exif.items():
labeled[TAGS.get(key)] = val
return labeled

def get_geotagging(exif):
if not exif:
raise ValueError("No EXIF metadata found")
geotagging = {}
for (idx, tag) in TAGS.items():
if tag == "GPSInfo":
if idx not in exif:
raise ValueError("No EXIF geotagging found")
for (key, val) in GPSTAGS.items():
if key in exif[idx]:
geotagging[val] = exif[idx][key]
return geotagging

def get_decimal_from_dms(dms, ref):
degrees = dms[0][0] / dms[0][1]
minutes = dms[1][0] / dms[1][1] / 60.0
seconds = dms[2][0] / dms[2][1] / 3600.0
if ref in ["S", "W"]:
degrees = -degrees
minutes = -minutes
seconds = -seconds
return round(degrees + minutes + seconds, 5)

def get_coordinates(geotags):
lat = get_decimal_from_dms(geotags["GPSLatitude"], geotags["GPSLatitudeRef"])
lon = get_decimal_from_dms(geotags["GPSLongitude"], geotags["GPSLongitudeRef"])
return (lat, lon)

exif = get_exif(fileIn)
exif_labeled = get_labeled_exif(exif)
print(exif_labeled["DateTime"])

geotags = get_geotagging(exif)
print(get_coordinates(geotags))

====Template Matching / Image Regocnition Using CV2 / OpenCV====
see [[Python - CV2]]

====Optical Character Recognition (OCR) ====
see [[Python - OCR]]

==Templates/Snippets==
===Retry on Exception===
for retry_no in range(MAX_RETRIES):
try:
if retry_no > 0:
logger.warning("retrying %d/%d...", retry_no + 1, MAX_RETRIES)
doit()
except json.JSONDecodeError:
logger.exception("JSONDecodeError")
if retry_no == MAX_RETRIES - 1:
# end of retries
raise

===Diff of 2 files===
import difflib

fh1 = p_file1.open("r", encoding="utf-8")
fh2 = p_file1.open("r", encoding="utf-8")
diff = difflib.ndiff(fh1.readlines(), fh2.readlines())
delta = "".join(line for line in diff if line.startswith(("+ ", "- ")))
print(delta)

===GPX parsing===
import gpxpy
import gpxpy.gpx
# Elevation data by NASA: see lib at https://github.com/tkrajina/srtm.py
fh_gpx_file = open(gpx_file_path, 'r')
gpx = gpxpy.parse(fh_gpx_file)
# Loops for accessing the data
for track in gpx.tracks:
for segment in track.segments:
for point in segment.points:
for waypoint in gpx.waypoints:
for route in gpx.routes:
for point in route.points:
# interesting properties of point / waypoint objects:
point.time
point.latitude
point.longitude
point.source
waypoint.name

===TypeHints===
from typing import Any, Dict, cast
creds = cast(dict[str, str], tomllib.load(f)) # type: ignore
o = cast(Dict[str, Any], tomllib.load(f)) # type: ignore
o["sap"] = cast(Dict[str, str], o["sap"])
o["settings"] = cast(Dict[str, str | int | bool], o["settings"])
o["settings"]["sleep_time"] = cast(int, o["settings"]["sleep_time"])

===Pylance/Pyright===
import tomllib
# shows warning: Import "xyz" could not be resolved
# fix by
import tomllib # pyright: ignore

===asserts function argument validation===
aus [https://bildungsportal.sachsen.de/opal/auth/RepositoryEntry/12518195218/CourseNode/94506982489539?0 Python Kurs von Carsten Knoll]
def eine_funktion(satz, ganzzahl, zahl2, liste):
if not type(satz) == str:
print "Datentpyfehler: satz"
return -1
if not isinstance(ganzzahl, int):
print "Datentpyfehler: ganzzahl"
return -2
if not isinstance(liste, (tuple, list)):
print "Datentpyfehler: liste"
return -3
# Kompakteste Variante (empfohlen):
assert zahl2 > 0, "Error: zahl2 ist nicht > 0" # Assertation-Error bei Nichterfuellung

def F(x):
if not isinstance(x, (float, int)):
msg = "Zahl erwartet, %s bekommen" % type(x)
raise ValueError(msg)
return x**2
better:
def F(x):
assert isinstance(x, (float, int)), "Error: x is not of type float or int"
return x**2

assert variant in [
"normal",
"gray",
"cannyedge",
], "Error: variant is not in 'normal', 'gray', 'cannyedge'"

===Exceptions===
see [https://medium.com/@saadjamilakhtar/5-best-practices-for-python-exception-handling-5e54b876a20]

====Catching====
Catch keyboard interrupt and do a "save exit"
try:
i = 0
while 1:
i += 1
print(i)
except KeyboardInterrupt:
print("stopped")

Catch '''all''' exceptions
try:
[...]
except Exception as e:
print("Exception: ", e)

====Raising Exceptions====
if resp.status_code != 200: # noqa: PLR2004
msg = f"Bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise ValueError(msg) from None

Custom Exceptions
try:
raise Exception("HiHo")

raise ValueError¶

===perl grep and map===
from [https://stackoverflow.com/questions/12845288/grep-on-elements-of-a-list]
def grep(list, pattern):
expr = re.compile(pattern)
return [elem for elem in list if expr.match(elem)]
or
filteredList = filter(lambda x: x < 7 and x > 2, unfilteredList)

def map(list, was, womit):
return list(map(lambda i: re.sub(was, womit, i), list))
# was = '.*"(\d+)".*'
# womit = r"\1"

===unit testing using pytest===
install via
pip install pytest
activate in vscode, see [https://code.visualstudio.com/docs/python/testing#_enable-a-test-framework]: To enable testing, use the Python: Configure Tests command on the Command Palette.

see https://docs.pytest.org/en/6.2.x/assert.html#assert

test_1.py:
import myLib # my custom lib to test

class TestClass:
def test_one(self):
x = "this"
assert "h" in x

def test_two(self):
assert myLib.multiply(1, 2) == 2

def test_three(self):
assert myLib.multiply(2, 2) == 2

====project structure====
=====simplest structore: test next to script=====
Alternative without tests dir:
./main.py
./helper.py
./helper_test.py
with helper_test.py:
from helper import fnc1

=====standalone model with src and tests dirs=====
for this dir structure
src/helper.py
tests/helper_test.py
helper_test.py needs
import sys
from pathlib import Path
sys.path.insert(0, (Path(__file__).parent.parent / "src").as_posix())

=====standalone model with tests dir=====
if the test files are placed inside a ./tests/ dir
./main.py
./helper.py
tests/helper_test.py
helper_test.py needs
sys.path.insert(0, Path(__file__).parent.parent.as_posix())
from helper import fnc1

====parametrize====
import pytest
@pytest.mark.parametrize(
("test_input", "expected"),
[("", None), ("PT30M", 30), ("PT3H", 3 * 60), ("PT2H30M", 2 * 60 + 30)],
)
def test_task_est_to_minutes(test_input: str, expected: int) -> None:
assert task_est_to_minutes(test_input) == expected

=====fixures: prepare and cleanup test_data=====
to automatically run before and after each testcase
@pytest.fixture(autouse=True)
def _setup_tests(): # noqa: ANN202
cache_prepare_lists()
cache_prepare_tasks()

yield

cache_cleanup_test_data()

====Test coverage report====
pip install pytest-cov
# console output
pytest --cov

# html report in coverage_report/index.html with details per file
pytest --cov --cov-report=html:coverage_report
to exclude a code block from coverage report, add
# pragma: no cover

=== Sleep / Wait for input ===
sleep for a while
import time
time.sleep(60)

wait for user input
input("press Enter to close")

===Suppress Warnings===
see [https://docs.python.org/3/library/warnings.html#temporarily-suppressing-warnings]
import warnings
warnings.filterwarnings("ignore", message=".*native_field_num.*not found in message.*")

===Logging===
====V4 minimal stdout logging====
import logging
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s")
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
logging.getLogger("httpx").setLevel(logging.WARNING)

# usage of variables and rounding of numbers. (%d also rounds floats)
LOGGER.info("file %s has %d lines with avg %.1f char/line", filename, lines, c_p_l)

====V3 simple logging====
# main file:
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(name)s\t%(message)s",
handlers=[
RotatingFileHandler(
Path(__file__).with_suffix(".log"),
maxBytes=10485760, # 10 MB = 10*1024*1024,
backupCount=1,
encoding="utf-8",
)
],
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(Path(__file__).stem)
logger.info("Start")

# other files:
import logging
logger = logging.getLogger(__name__)

====V2: rotating file and STDOUT====
# 1. setup
import logging
from logging.handlers import RotatingFileHandler

logfile = "myApp.log"
maxBytes = 20 * 1024 * 1024
backupCount = 5
loglevel_console = logging.INFO
loglevel_file = logging.DEBUG

# create logger
logger = logging.getLogger("root")
logger.setLevel(loglevel_file)

# console handler
ch = logging.StreamHandler()
ch.setLevel(loglevel_console)

# rotating file handler
# fh = logging.FileHandler(logfile)
fh = RotatingFileHandler(logfile, maxBytes=maxBytes, backupCount=backupCount)
fh.setLevel(loglevel_file)

# create formatter and add it to the handlers
# %(name)s = LoggerName, %(threadName)s = TreadName
formatter = logging.Formatter(
"%(asctime)s - %(levelname)s - %(name)s - %(threadName)s - %(message)s "
)
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(ch)
logger.addHandler(fh)

logger.debug("DebugMe")
logger.info("Starting")
logger.warning("Attention")
logger.error("Something went wrong")
logger.critical("Something seriously went wrong ")

# 2. in other files/modules now use
import logging

logger = logging.getLogger(__name__)
logger.info("text")

...
except Exception as e:
logger.exception("Unhandeled exception")
quit()

====V1: Simple====
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(message)s", # \t%(name)s
filename=Path(__file__).with_suffix(".log"), # uncomment to switch to stdout
filemode="a",
)
logger = logging.getLogger(__name__)
logger.debug("Some text")
logger.info("Some text")
logger.warning("Some text")
logger.error("Some text")
logger.critical("Some text")
try:
...
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise

===Process Bar===
see [https://tqdm.github.io tqdm]
from tqdm import tqdm
for i in tqdm(range(10000)):
....
or
with tqdm(total=total_iterations, desc="Processing") as progress_bar:
...
progress_bar.update(1)

===CGI Web development===
# Print necessary headers.
print("Content-Type: text/html")
print()

# errors and debugging info to browser
import cgitb
cgitb.enable()

Access URL or Form Parameters
# V2 from https://www.tutorialspoint.com/python/python_cgi_programming.htm
import cgi
form = cgi.FieldStorage()
username = form.getvalue('username')
print(username)

# V1
import sys
import urllib.parse
query = os.environ.get('QUERY_STRING')
query = urllib.parse.unquote(query, errors="surrogateescape")
d = dict(qc.split("=") for qc in query.split("&"))
print(d)

====CGI Backend Returning JSONs====
#!/usr/local/bin/python3.6
# -*- coding: utf-8 -*-

import cgi
import json

# Print necessary headers.
print("Content-type: application/json")
print()

def get_form_parameter(para: str) -> str:
"asserts that a given parameter is set and returns its value"
value = form.getvalue(para)
assert value, f"Error: parameter {para} missing"
assert value != "", f"Error: parameter {para} missing"
return value

response = {}
response['status'] = "ok"

try:
action = get_form_parameter("action")
response['action'] = action
if action == "myAction":
...

except Exception as e:
response['status'] = "error"
d = {"type": str(type(e)), "text": str(e)}
response["exception"] = d

finally:
print(json.dumps(response))

==Databases==
===PostgreSQL===
====Improved helper function====
"""Helper functions for DB access."""

import datetime as dt
import logging

import psycopg2
import psycopg2.extras
from creds_db import credentials as creds_db

# Configure logging
logger = logging.getLogger(__name__)

connection: psycopg2.extensions.connection = None # type: ignore
cursor_query: psycopg2.extensions.cursor = None # type: ignore
cursor_write: psycopg2.extensions.cursor = None # type: ignore

def connect() -> (
tuple[
psycopg2.extensions.connection,
psycopg2.extensions.cursor,
psycopg2.extensions.cursor,
]
):
"""Connect to the database."""
try:
connection = psycopg2.connect(**creds_db)
cursor_query = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)
cursor_write = connection.cursor()
logger.info(
"Connected to database %s on host %s",
creds_db["database"],
creds_db["host"],
)
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise
else:
return connection, cursor_query, cursor_write

def disconnect() -> None:
"""Disconnect from the database."""
try:
if cursor_query:
cursor_query.close()
if cursor_write:
cursor_write.close()
if connection:
connection.close()
logger.info("Disconnected from the database")
except psycopg2.DatabaseError:
logger.exception("Error during disconnection: %s")

def exists(url: str, valid_hours: float) -> bool | None:
"""
Check if a record exists.

Returns
-------
None for missing record
True for valid record
False for expired record

"""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=valid_hours)
sql = """
SELECT delete_at > %s AS "valid"
FROM my_table
WHERE url = %s LIMIT 1;
"""
cursor_query.execute(sql, (delete_at, url))
res = cursor_query.fetchone()
return res[0] if res else None

def query1(url: str) -> dict | None:
"""Execute a query."""
sql = """
SELECT *
FROM my_table
WHERE url = %s LIMIT 1
"""
cursor_query.execute(sql, (url,))
return cursor_query.fetchone() # type: ignore

def insert(url: str, response: str, delete_in_hours: float) -> None:
"""Insert a record."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=delete_in_hours)
sql = """
INSERT INTO my_table (url, response, delete_at)
VALUES (%s, %s, %s);
"""
cursor_write.execute(sql, (url, response, delete_at))
connection.commit()

def delete(url: str) -> None:
"""Delete a record."""
sql = """
DELETE FROM my_table
WHERE url = %s;
"""
cursor_write.execute(sql, (url,))
connection.commit()

def delete_expired(in_hours: float) -> None:
"""Delete records that expire in less than in_hours from now."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=in_hours)

sql = """
DELETE FROM my_table
WHERE delete_at < %s;
"""
cursor_write.execute(sql, (delete_at,))
connection.commit()

connection, cursor_query, cursor_write = connect()

====Basics====
import psycopg2
import psycopg2.extras

credentials = {
"host": "localhost",
"port": 5432,
"database": "myDB",
"user": "myUser",
"password": "myPwd",
}
connection = psycopg2.connect(**credentials)
cursor = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)

l_bind_vars = [["A1", "A2"], ["B1", "B2"]]

sql = """
SELECT * FROM myTable
WHERE 1=1
AND status NOT IN ('CLOSED')
AND ColA = %s
AND ColB = %s
ORDER BY created DESC
"""
cursor.execute(sql, l_bind_vars)
d_data = dict(cursor.fetchone())

====export result to csv file====
sql1 = "SELECT * FROM table"
sql2 = "COPY (" + sql1 + ") TO STDOUT WITH CSV HEADER DELIMITER '\t'"
with open("out.csv", "w") as file:
cursor.copy_expert(sql2, file)

====ReadConfigFile to connect to PostgreSQL====
from [http://www.postgresqltutorial.com/postgresql-python/connect/]
database.ini
[postgresql]
host=dbhost
port=5432
database=dbname
user=dbuser
password=dbpass

from configparser import ConfigParser

def config(filename="database.ini", section="postgresql"):
parser = ConfigParser()
parser.read(filename)
# get section, default to postgresql
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception(
"Section {0} not found in the {1} file".format(section, filename)
)
return db

main.py
import psycopg2
from config import config

def connect():
"""Connect to the PostgreSQL database server"""
conn = None
try:
params = config() # read connection parameters
print("Connecting to the PostgreSQL database...")
conn = psycopg2.connect(**params)
cur = conn.cursor() # create a cursor
print("PostgreSQL database version:")
cur.execute("SELECT version()") # execute a statement
db_version = cur.fetchone()
print(db_version)
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
print("Database connection closed.")

if __name__ == "__main__":
connect()

===SQL Lite / SQLite===
see page [[SQLite]]

===InfluxDB===
see [[InfluxDB]]

==Internet Access==
===Send E-Mails===
see [[Python - eMail]]

===Download data using browser UA / REST===
import requests

headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0",
}
resp = requests.get(
url,
headers=headers,
timeout=3, # timeout in sec, requests should always have a timeout!
# timeout=(1,3), # 1s connect-timout, 3s read-timeout
)
if resp.status_code != 200: # noqa: PLR2004
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download only if cache is too old====
import time
from pathlib import Path
import requests

def fetch_url_or_cache(file_cache: Path, url) -> str:
"""Fetch URL and cache in a file."""
if check_cache_file_available_and_recent(
file_path=file_cache, max_age=3600, verbose=False
):
with file_cache.open(mode="r", encoding="utf-8") as fh:
s = fh.read()
else:
s = fetch(url=url)
with file_cache.open(mode="w", encoding="utf-8", newline="\n") as fh:
fh.writelines(s)
return s

def check_cache_file_available_and_recent(
file_path: Path,
max_age: int = 3500,
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = False
if file_path.exists() and (time.time() - file_path.stat().st_mtime < max_age):
cache_good = True
return cache_good

# or
def check_cache_file_available_and_recent_verbose(
file_path: Path, max_age: int = 3600, *, verbose: bool = False
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = True
if not file_path.exists():
if verbose:
print(f"No Cache available: {file_path}")
cache_good = False
if cache_good and time.time() - (time.time() - file_path.stat().st_mtime < max_age):
if verbose:
print(f"Cache too old: {file_path}")
cache_good = False
return cache_good

def fetch(url) -> str:
"""Fetch URL."""
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0", # noqa: E501
}
resp = requests.get(url, headers=headers, timeout=5)
if resp.status_code == 200: # noqa: PLR2004
return resp.content.decode("ascii")
else: # noqa: RET505
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download using urllib.request====
from urllib.request import urlopen

url = "https://pomber.github.io/covid19/timeseries.json"
with urlopen(url) as response: # noqa: S310
response_text = response.read()

===HTML parsing and extracting of elements===
V2: via BeautifulSoup
from bs4 import BeautifulSoup # pip install beautifulsoup4

soup = BeautifulSoup(cont, features="html.parser")
my_element = soup.find("div", {"class": "user-formatted-inner"})
my_body = my_element.prettify()
# my_body = my_element.encode()
# my_body = str(my_element)

V1: via lxml and xpath
from lxml import html
import requests

page = requests.get(url)
tree = html.fromstring(page.content)
tbody_trs = tree.xpath("//*/tbody/tr")
l_rows = []
for tr in tbody_trs:
l_columns = []
if len(tr) != 15:
continue
for td in tr:
l_columns.append(td.text_content())
l_rows.append(list(l_columns))

===HTML entities to unicode===
import html
cont = html.unescape(cont)

==GUI Interactions==
===Take Screenshot===
import pyautogui # (c:\Python\Scripts\)pip install pyautogui
# pyautogui does only support screenshots on monitor #1
...
screenshot = pyautogui.screenshot()
# screenshot = pyautogui.screenshot(region=(screenshotX,screenshotY, screenshotW, screenshotH))
screenshot = np.array(screenshot)
# Convert RGB to BGR
screenshot = screenshot[:, :, ::-1].copy()

===Mouse Actions===
def clickIt(x,y,key="") :
x0, y0 = pyautogui.position()
if key != "": # crtl, shift
pyautogui.keyDown(key)
pyautogui.moveTo(x, y, duration=0.2)
pyautogui.click(x=x , y=y, button='left', clicks=1, interval=0.1)
if key != "": # crtl, shift
pyautogui.keyUp(key)
pyautogui.moveTo(x0, y0)

===Web Automation===
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time
import glob

class StravaUserMapDL():
def __init__(self):
self.driver = webdriver.Firefox()

def login(self):
driver = self.driver
url = "https://www.somewebpage.com"
email = "myemail"
password = "mypassword"
driver.get(url)

title = driver.title
urlIs = driver.current_url
cont = driver.page_source # as string
FILE = open(filename,"w") # w = overWrite file ; a = append to file
FILE.write(cont)
FILE.close()

# handle login if urlIs != url
if (urlIs != url):
# activate checkbox 'remember_me'
elem = driver.find_element_by_id('remember_me')
if (elem.is_selected() == False):
elem.click()
assert elem.is_selected() == True
elem = driver.find_element_by_id('email')
elem.send_keys(email)
elem = driver.find_element_by_id('password')
elem.send_keys(password)
elem.send_keys(Keys.RETURN)
# Wait until login pages is replaced by real page
urlIs = driver.current_url
while (urlIs != url):
time.sleep(1)
urlIs = driver.current_url
print (urlIs)

# results = driver.find_elements_by_class_name('following')
# results = driver.find_elements_by_tag_name('li')

# print(results[0].text)
assert (urlIs == url)

===Unit Tests using Web Automation===
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time

class PythonOrgSearch(unittest.TestCase):
# def __init__(self,asdf):
# self.driver = webdriver.Firefox()

def setUp(self):
print ("setUp")
# headless mode:
# opts = Options()
# opts.set_headless()
# assert opts.headless # Operating in headless mode
# self.driver = webdriver.Firefox(options=opts)

self.driver = webdriver.Firefox()

def test_search_in_python_org(self):
driver = self.driver
driver.get("http://www.python.org")
self.assertIn("Python", driver.title)
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
print ("fertig: python_org")

def tearDown(self):
print ("tearDown")
print ("close Firefox")
self.driver.close() # close tab
self.driver.quit() # quit browser
# os._exit(1) # exit unittest without Exception

if __name__ == "__main__":
try:
unittest.main()
except SystemExit as e:
os._exit(1)

==Cryptography and Hashing==
====Hashing via SHA256====
def gen_SHA256_string(s: str) -> str:
m = hashlib.sha256()
m.update(s.encode("ascii"))
return m.hexdigest()

====Hashing via MD5====
(MD5 is not secure, better use SHA256)
def gen_MD5_string(s: str) -> str:
m = hashlib.md5()
m.update(s.encode("ascii"))
return m.hexdigest()

====Password hashing via bcrypt====
import bcrypt
pwd = 'geheim'
pwd = pwd.encode("utf-8")
# or
pwd = b'geheim'

hashed = bcrypt.hashpw(pwd, bcrypt.gensalt())
if bcrypt.checkpw(pwd, hashed):
print("It Matches!")
print(hashed.decode("utf-8"))

To use version 2a instead of 2b (default):
bcrypt.gensalt(prefix=b"2a")

==Multiprocessing, subprocesses and Threading==
see [[Python_-_Multithreading]] as well

use processes for CPU limited work 
use threads for I/O limited work

===Simple single process===
import subprocess
process = subprocess.run(["sudo", "du", "--max-depth=1", mydir], capture_output=True, text=True)
print (process.stdout)
old, depricated way:
os.system( "gnuplot " + gpfile)

===Multiprocessing===
see [[Python - Multithreading]] as well

V2 using pool and starmap
import multiprocessing
import os

def worker(i: int, s: str) -> list:
result = (i, s, os.getpid())
return result

if __name__ == "__main__":
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n" + str(i))
l_pile_of_work.append((tup))
# gen pool of processes
num_processes = min(multiprocessing.cpu_count(), len(l_pile_of_work))
pool = multiprocessing.Pool(processes=num_processes)
# start processes on pile of work
l_results_unsorted = pool.starmap(
func=worker, iterable=l_pile_of_work # each item is a list of 2 parameters
)

# or if only one parameter:
# l_results_unsorted = pool.map(doit_de_district, l_pile_of_work)

l_results = sorted(l_results_unsorted) # sort by i

V1
import subprocess
l_subprocesses = [] # queue list of subprocesses
max_processes = 4

def process_enqueue(new_process_parameters):
global l_subprocesses
# wait for free slot
while len(l_subprocesses) >= max_processes:
process_remove_finished_from_queue()
time.sleep(0.1) # sleep 0.1s
process = subprocess.Popen(new_process_parameters,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
l_subprocesses.append(process)

def process_remove_finished_from_queue():
global l_subprocesses
i = 0
while i <= len(l_subprocesses) - 1:
process = l_subprocesses[i]
if process.poll != None: # has already finished
process_print_output(process)
l_subprocesses.pop(i)
else: # still running
i += 1

def process_print_output(process):
"""waits for process to finish and prints process output"""
stdout, stderr = process.communicate()
if stdout != '':
print(f'Out: {stdout}')
if stderr != '':
print(f'ERROR: {stderr}')

def process_wait_for_all_finished():
global l_subprocesses
for process in l_subprocesses:
process_print_output(process)
l_subprocesses = [] # empty list of done subprocesses

process_enqueue(l_parameters1)
...
process_enqueue(l_parameters999)
process_wait_for_all_finished()

===Threading===
import threading
import queue
import os
import time

def worker(q_work: queue.Queue, results: dict):
while not q_work.empty():
i, s = q_work.get()
time.sleep(.1)
result = (i, s, os.getpid())
results[i] = result
q_work.task_done()

if __name__ == '__main__':
d_results = {} # threads can write into dict
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n"+str(i))
l_pile_of_work.append((tup))
# convert list of work to queue
q_pile_of_work = queue.Queue(
maxsize=len(l_pile_of_work)) # maxsize=0 -> unlimited
for params in l_pile_of_work:
q_pile_of_work.put(params)
# gen threads
num_threads = 100
l_threads = [] # List of threads, not used here
for i in range(num_threads):
t = threading.Thread(name='myThread-'+str(i),
target=worker,
args=(q_pile_of_work, d_results),
daemon=True)
l_threads.append(t)
t.start()
q_pile_of_work.join() # wait for all threas to complete
l_results_unsorted = d_results.values()
l_results = sorted(l_results_unsorted) # sort by i

===asyncio — Asynchronous I/O===
see https://docs.python.org/3/library/asyncio-task.html
import asyncio
import time

# basics
# task = asyncio.create_task(coro())
# Wrap the coro coroutine into a Task and schedule its execution. Return the Task object.

# sleep
# await asyncio.sleep(1)

# Running Tasks Concurrently and gathers the return values in list L
# L = await asyncio.gather(coro(x1,y1), coro(x2,y2), coro(x3,y3))

async def say_after(delay, what):
# Coroutines declared with the async/await syntax
await asyncio.sleep(delay)
print(what)

async def main():
# The asyncio.create_task() function to run coroutines concurrently as asyncio Tasks.
task1 = asyncio.create_task(
say_after(1, 'hello'))

task2 = asyncio.create_task(
say_after(1, 'world'))

print(f"started at {time.strftime('%X')}")

# Wait until both tasks are completed
await task1
await task2

print(f"finished at {time.strftime('%X')}")

asyncio.run(main())

==Pandas==
see [[Pandas]]

==Matplotlib==
see [[Matplotlib]]

== GUI via tkinter==
import tkinter as tk # no need to install via pip

class App(tk.Tk):
def __init__(self):
super().__init__()
self.title("robot-CClicker")
self.geometry("200x200+0+1080")
self.resizable(width=False, height=False)

self.l_buttons = []
self.__create_widgets()

def __create_widgets(self):
self.btn_click500 = tk.Button(
master=self,
text="500 clicks",
width=20,
command=lambda: self.clickBigCookie(500),
)
self.l_buttons.append(self.btn_click500)

for button in self.l_buttons:
button.pack(anchor=tk.W)

def clickBigCookie(self, num):
# TODO: the disabling of the buttons is not working
for button in self.l_buttons:
button["state"] = tk.DISABLED
helper.clickIt(self.posBigCockie[0], self.posBigCockie[1], num=num)
for button in self.l_buttons:
button["state"] = tk.NORMAL

if __name__ == "__main__":
app = App()
app.mainloop()

==Protobuf==
===convert .proto file to python class===
see https://www.datascienceblog.net/post/programming/essential-protobuf-guide-python/
protoc my_message.proto --python_out ./

===read/decode protobuf message===
parse message from file
import machine_message_pb2

with open("out.bin", "rb") as f:
my_message = my_message_pb2.my_message()
my_message.ParseFromString(f.read())
print(machine_message)

===create/encode protobuf message===
import machine_message_pb2

my_message = my_message_pb2.my_message()
my_message.data.field1 = 1
my_message.data.field2 = "asdf"

with open("out.bin", "wb") as f:
f.write(my_message.data.SerializeToString())

==venv / virtual environments==
pip install --upgrade pip
python -m venv .venv --prompt $(basename $(pwd))
source .venv/bin/activate
pip install -r requirements.txt
...
deactivate

==Pydantic data validation==
===Function input parameter validation===
see https://docs.pydantic.dev/usage/validation_decorator/
@validate_arguments
def sum(x: int, y: float) -> float:
print(x, y)
return x + y

print(sum(1.1, 1.1))
# -> 2.1

==Read config file==
===TOML===
see https://learnxinyminutes.com/docs/toml/

minimal example:
config.toml
# SAP API endpoint
[general]
sleep_time = 60
use_proxy = true
tool.py
try:
import tomllib # comes with python3.11
except ModuleNotFoundError:
import tomli as tomllib # pip install tomli
with open("config.toml", "rb") as f:
o = tomllib.load(f) # type: ignore
if o["settings"]["use_proxy"]:
...
for validation, see https://realpython.com/python-toml/

====TOML Write====
In for deployments I sometime want to modify a TOML config file for production, based on the dev version.
import tomli_w # pip install tomli-w
p_in = Path(__file__).parent.parent / ".streamlit/config.toml"
p_out = p_in.parent / "config-prod.toml"
with p_in.open("rb") as fh:
o = tomllib.load(fh)
o["server"]["fileWatcherType"] = "none"
del o["server"]["address"]
with p_out.open("wb") as fh:
tomli_w.dump(o, fh)

===ConfigParser: INI Config File Reading===
better use TOML

Config.ini
[Section1]
Cursor = 205E18
Grandma = 18E18
Farm = 11E18
Mine = 514E18
Factory = 155E18
test.py
from configparser import ConfigParser

config = ConfigParser(
interpolation=None
) # interpolation=None -> treats % in values as char % instead of interpreting it
config.read("Config.ini", encoding="utf-8")

print(config.getint("Section1", "key1"))
print(config.getfloat("Section1", "key2"))
print(config.get("Section1", "key3"))

for sec in config.sections():
d_settings = {}
for key in config.options(sec):
value = config.get(sec, key)
d_settings[key] = value
print("%15s : %s" % (key, value))

==VCard / vcf==
import codecs
import vobject # pip install vobject

obj = vobject.readComponents(codecs.open(file_in, encoding="utf-8").read()) # type: ignore
contacts: list[vobject.base.Component] = list(obj) # type: ignore

card = contacts[0]

if "bday" not in card.contents:
continue

# bday: remove 'VALUE': ['DATE']
card.contents["bday"][0].params = {} # type: ignore

# remove all fields but "bday", "n"
for key in card.contents.copy(): # loop over copy, to allow for deleting keys
if key not in ("n", "bday"):
del card.contents[key]

# recreate fn based on n
card.add("fn")
n = card.contents["n"][0]
fn = f"{n.value.given} {n.value.additional} {n.value.family}" # type: ignore
fn = re.sub(r"\s+", " ", fn)
card.fn.value = fn # type: ignore

with open("out.vcf", mode="w", encoding="utf-8", newline="\n") as fhOut:
fhOut.write(card.serialize())

==Sentry Exception monitoring==
see [[Sentry]]
==MQTT==
#!/usr/bin/env python3.12
"""Read power data from Tasmota device via MQTT."""

import json
from typing import Any

import paho.mqtt.client as mqtt
from mqtt_credentials import hostname, password, port, username
from paho.mqtt.reasoncodes import ReasonCode

def on_connect(
mqtt_client: mqtt.Client,
userdata: Any,
flags: dict[str, int],
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client connects MQTT broker.""" # noqa: D401
if reason_code == 0:
print("Connected to MQTT")
# print(f"MQTT protocol version: {mqtt_client._protocol}")

# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
mqtt_client.message_callback_add("tele/tasmota_MT681/SENSOR", on_message)
mqtt_client.subscribe([("tele/tasmota_MT681/SENSOR", 0)])

else:
print(f"Connection to MQTT broker failed. Error: {reason_code}")

def on_disconnect(
mqtt_client: mqtt.Client,
userdata: Any,
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client is disconnected from the MQTT broker.""" # noqa: D401
if reason_code > 0:
print(f"Unexpected disconnection. Error: {reason_code}")

def on_message(
mqtt_client: mqtt.Client,
userdata: Any,
message: mqtt.MQTTMessage,
) -> None:
"""Callback when a Tasmota message is received from the MQTT broker.""" # noqa: D401
s = message.payload.decode()
data = json.loads(s)
# {'Time': '2024-03-16T06:44:08', 'MT681': {'Total_in': 16.4396, 'Power_cur': 80, 'Total_out': 14.6403}} # noqa: E501

total_i = data["MT681"]["Total_in"]
total_o = data["MT681"]["Total_out"]

print(f"In:\t{total_i}\nOut:\t{total_o}")

mqtt_client.disconnect()
mqtt_client.loop_stop()

try:
# Create an MQTT client instance
mqtt_client: mqtt.Client = mqtt.Client(
mqtt.CallbackAPIVersion.VERSION2, "Python-Client-1"
) # type: ignore
mqtt_client.username_pw_set(username, password)

mqtt_client.on_connect = on_connect

# Connect to the MQTT broker
mqtt_client.connect(hostname, port, keepalive=60)

# NOT: while True, as that eats the CPU !!!
mqtt_client.loop_forever()
except KeyboardInterrupt:
print("KeyboardInterrupt, disconnecting from MQTT broker.")
mqtt_client.disconnect()
mqtt_client.loop_stop()

==Caching Functions==
from functools import lru_cache

@lru_cache(maxsize=128) # None equals to @cache declarator
def fnc(n:int):

==Performance/Resources==
===RAM/Memory consumption/bottleneck===
import tracemalloc
tracemalloc.start()

# ...
# Code
# example:
lst = list(range(1_000_000))
# ...

max_bytes = tracemalloc.get_traced_memory()[0]
print(f"{round(max_bytes / 10**6)}MB")

tracemalloc.stop()

===Profiling / Count and Runtime per Function===
call_stats = {}

def track_function_usage(func: Callable) -> Callable:
"""Annotation for gathering runtime statistics."""

@wraps(func)
def wrapper(*args: tuple, **kwargs: dict): # noqa: ANN202
start_time = time.time()
result = func(*args, **kwargs) # Call the original function
end_time = time.time()
call_stats[func.__name__]["calls"] += 1
call_stats[func.__name__]["total_time"] += end_time - start_time
return result
return wrapper

==JWT Token==
try:
decoded_token = jwt.decode(token, options={"verify_signature": False})
except jwt.ExpiredSignatureError:
msg = "Token has expired."
logger.exception(msg)
except jwt.InvalidTokenError:
msg = "Invalid token."
logger.exception(msg)

==Streamlit==
see https://github.com/entorb/streamlit-examples for a collection of my examples

===Minimum Example===
# src/app.py
from pathlib import Path
import streamlit as st
from streamlit.logger import get_logger
logger = get_logger(Path(__file__).stem)
logger.info("Start")
st.set_page_config(page_title="AppTitle", page_icon=None, layout="wide")
st.title("MyPageTitle")
st.header("MyHeader")
st.subheader("MySubHeader")
sel_param = st.selectbox("Parameter", ("count", "sum"))

# .streamlit/config.toml
[browser]
gatherUsageStats = false

[server]
port = 8501

# run it
streamlit run src/app.py

====Reduce Logging for K8s Deployed Docker Containers====
# reduce log output
for logger_name in [
"streamlit.runtime.caching.cache_utils",
"streamlit.runtime.caching.storage.in_memory_cache_storage_wrapper",
"streamlit.runtime.runtime",
"urllib3.connectionpool",
]:
get_logger(logger_name).setLevel(logging.WARNING)

===Chart via Altair===
====Line Chart====
chart = st.line_chart(df["value"])
# corresponds to
import altair as alt
c = (
alt.Chart(df)
.mark_line()
.encode(
x=alt.X("created", title=None),
y=alt.Y("value", title=None),
)
)
st.altair_chart(c, use_container_width=True)

====Bar Chart====
chart = (
alt.Chart(df)
.mark_bar()
.encode(
x=alt.X("yearmonth(date):T", title="Month"),
y=alt.Y(f"count:Q", title=None),
color="status:N",
tooltip=["yearmonth(date):T", "status:N", "cnt:Q"],
)
# .properties(width=600, height=400)
)
st.altair_chart(chart, use_container_width=True)

:T stands for Temporal. It indicates that the field contains date or time values.
:N stands for Nominal. It indicates that the field contains categorical data, which represents discrete categories or labels.
:Q stands for Quantitative. It indicates that the field contains numerical data that can be measured and compared.

===Excel Download===
def excel_download_buttons(df: pd.DataFrame, file_name: str = "export.xlsx") -> None:
"""Show prepare data and download buttons."""
if st.button(label="Click to prepare download"):
buffer = io.BytesIO()
with pd.ExcelWriter(buffer, engine="xlsxwriter") as writer:
df.to_excel(writer, sheet_name="Sheet1", index=False)
writer.close()

st.download_button(
label="Download data as Excel",
data=buffer,
file_name=file_name,
mime="application/vnd.ms-excel",
)

==UV Package Manager==
# create/update lockfile
uv lock --upgrade
# sync venv
uv sync --upgrade
# extract package info to requirements.txt
uv export --format requirements.txt --no-dev --no-hashes -o requirements.txt

Update Python version for my current project
uv venv --python=3.13.7

===Update UV===
uv self update
# or
brew update && brew upgrade uv

===Update Python Versions===
uv python upgrade

===UV + Docker on readOnlyRootFilesystem===
Example 1: https://github.com/SWE-Group-23/CM22007---Source/blob/10d02004f3a4208f7b30d37a0a87e89532a23575/create-service.sh#L117

Dockerfile
WORKDIR /app
COPY . .

ENV CASS_DRIVER_NO_EXTENSIONS=1

RUN --mount=type=cache,target=/root/.cache/uv \\
--mount=type=bind,source=uv.lock,target=uv.lock \\
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \\
uv sync --frozen --no-install-project

RUN uv pip install -e shared/

RUN addgroup -S group1 && adduser -S user1 -G group1 -u 1000
RUN chown -R user1:group1 /app
USER user1:group1

CMD ["uv", "run", "main.py"]

Deployment
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /home/user1/.cache/uv
name: uv
- mountPath: /tmp
name: temp

Example 2: [https://hynek.me/articles/docker-uv/ Production-ready Python Docker Containers with uv]

Dockerfile (without the comments)
ENV UV_LINK_MODE=copy \
UV_COMPILE_BYTECODE=1 \
UV_PYTHON_DOWNLOADS=never \
UV_PYTHON=python3.12 \
UV_PROJECT_ENVIRONMENT=/app

RUN --mount=type=cache,target=/root/.cache \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync \
--locked \
--no-dev \
--no-install-project

COPY . /src
WORKDIR /src
RUN --mount=type=cache,target=/root/.cache \
uv sync \
--locked \
--no-dev \
--no-editable

== Vulture: Search for dead code ==
see [https://github.com/jendrikseipp/vulture GitHub]

pyproject.toml
[tool.vulture]
paths = ["src"]
# exclude = ["src/models.py"]
ignore_names = [
"LOGGER",
]
# ignore_decorators = []

.pre-commit-config.yaml
# Vulture: find dead code
- repo: https://github.com/jendrikseipp/vulture
rev: v2.16
hooks:
- id: vulture
pass_filenames: false

run
pip install vulture
vulture src/

uv add --dev vulture
uv run vulture src/

Python

2026-04-13T12:35:17Z

Torben: /* Vulture: Search for dead code */

[[Category:Coding]][[Category:Python]]

==Getting Started==
===Install===
====Python====
Best use [https://docs.astral.sh/uv/ UV] for managing python and package versions.

* for Windows: get and install Python from https://www.python.org
* for MacOS: use [https://github.com/pyenv/pyenv?tab=readme-ov-file#a-getting-pyenv pyenv]
brew install xz
brew install pyenv

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[ [ -d $PYENV_ROOT/bin ] ] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init - zsh)"' >> ~/.zshrc
exec "$SHELL"

pyenv install --list
pyenv install 3.13.5
pyenv global 3.13.5

vim ~/.zshrc
# add
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi

====Editor: Visual Studio Code====
excellent and free source-code editor that supports many languages.
* get and install from https://code.visualstudio.com

See Wickie page [[Visual_Studio_Code]] for general setup, extension, config...

Python Extensions
* Python
* Pylance
* [https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff Ruff Formater and Linter]
* [https://marketplace.visualstudio.com/items/?itemName=tamasfe.even-better-toml Even Better TOML]
Settings (CTRL + ,)
* Extensions -> Python -> Formatting: Provider = black
* Text Editor -> Editor: Format On Save
* Text Editor -> Files:Eol -> \n
* Linter: Ruff
Settings in settings.json
"python.analysis.completeFunctionParens": true,
"python.analysis.autoImportCompletions": true,
"python.analysis.inlayHints.functionReturnTypes": true,
"python.analysis.typeCheckingMode": "strict",
// Ruff
"[python]": {
"editor.defaultFormatter": "charliermarsh.ruff",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.fixAll": "explicit",
"source.organizeImports.ruff": "explicit"
},
},

Run your code
CTRL + F5 : run
F5 : run in debugger

====Code Formatter Ruff====
use software for handling the code formatting like "ruff" or "black", where Ruff is much faster and additionally provides linting.
pip install ruff
than activate in editor like vs code

===My Template===
see latest version at https://github.com/entorb/template-python

"""Main file."""
# by Torben Menke https://entorb.net
import logging
from pathlib import Path

def init_logging() -> None:
"""Initialize logging."""
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(
level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s"
)
logging.getLogger("httpx").setLevel(logging.WARNING)

init_logging()
LOGGER = logging.getLogger(__name__)

def main():
LOGGER.info("Moin Moin")

if __name__ == "__main__":
main()

"""Other file"""
import logging
LOGGER = logging.getLogger(__name__)

===Compile to .exe===
pip install pyinstaller
pyinstaller --onefile --console your.py
# for Excel and Matplotlib these options are required
--hidden-import=openpyxl --hidden-import=matplotlib --hidden-import pandas.plotting._matplotlib
([[Python - py2exe]] is deprecated)

==Basics==
=== Naming Conventions===
[https://google.github.io/styleguide/pyguide.html Google Python Style Guide]:
module_name, package_name, ClassName, method_name, ExceptionName, function_name, GLOBAL_CONSTANT_NAME, global_var_name, instance_var_name, function_parameter_name, local_var_name

====Doc Strings====
It is best practice to start a file with a docstring:
"""My Script"""
This can be accessed linke this:
title = __doc__[:-1] # type: ignore

===Installing packages===
python -m pip install --upgrade pip

pip install somemodule
# or
pip3 install somemodule
# or read from file
pip install -r requirements.txt

# uninstall
pip uninstall somemodule

# using a web proxy
# set proxy for windows cmd session
SET HTTPS_PROXY=http://myProxy:8080
(afterwards --proxy setting below no longer required
or
pip install --proxy http://myProxy:8080 somemodule

# list outdated packages
pip list --outdated

# update package
pip install --upgrade pyinstaller

# updating all via Windows Powershell (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | %{$_.split('==')[0]} | %{pip install --upgrade $_}

# updating all via Bash (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install --upgrade

# downgrade
pip install --upgrade pandas==1.2.4

====Oneline Updater for requirements.txt====
https://pkgui.com/pip

====update packages====
pip install pip-review
pip-review --auto

====generate requirements.txt====
pip install pipreqs
pipreqs ./

===Loops===
ATTENTION: The loops do not create a new variable scope. Only functions and modules introduce a new scope!
for p in all_files:
print(p)

for i, p in enumerate(all_files):
print(i, p)

for i in range(10):
print(i)
# del i

while i <= 100:
i += 1
...
if sth1:
continue # start next loop
if sth2:
break # exit loop

# inline if (requires a dummy else):
print("something") if sth else 0

===Math===
see [[Python - Math]] for linear regression

Modulo
15 % 4
# > 3

Integer Division
17 // 4
# > 4

====Random====
import random
random.randint(1000000, 9999999)

==Basic Objects==
===Variables===
del var # delete / undef a variable
var = None # sets to null

# check if variable is defined
if "var" in locals():
pass
# for object oriented projects:
if "var" in self.__dict__:
pass

# Access global variables in functions
var = 123

def test() -> None:
global var # point to global instead of creation of local var
var = 321

===Strings===
# num <-> str
s = str(i) # int to string
f = float(s) # str -> float
i = int(s)
str(round(f, 1)) # round first
# tests
s.isdigit() # 0-9
# note isdecimal() does also not match '1.1'

# printf: 1 digit
s = f"{value:0.1f}"
s = "%0.1f" % value

====Modify Strings====
# get string from prompt
s = input("Enter Text: ")

s = s.strip() # trim spaces from both sides, rstrip for right only
s = s.lower() # lower case
s = s.upper() # upper case
s = s.title() # upper case for first char of word

# upper case first letter of each word and also removes multiple and trailing spaces
import string
s = string.capwords(s)

# replace
s.replace(x, y)

# trim whitespaces from left and right
s.strip()

# replace all (multiple) whitespaces by single space ' '
s = " ".join(s.split())

# generate key value pairs from dict
# key1=value1&key2=value2
param_str = "&".join("=".join(tup) for tup in dict.items())

# repeat string multiple times
s * 5 # = s+s+s+s+s

====Substrings====
# find a substring:
if x in s: # True / False
pass
if len(s) > 0:
pass

# handling substrings
a = "abcd"
b = a[:1] + "o" + a[2:]
# > 'aocd'

s = "Hello there !bob@"
i1 = s.find("!") + 1
i2 = s.find("@")
substr = s[i1:i2]

def substr_between(s: str, s1: str, s2: str) -> str:
assert s1 in s, f"E: can't find '{s1}' in '{s}'"
assert s2 in s, f"E: can't find '{s1}' in '{s}'"
i1 = s.find(s1) + len(s1)
i2 = s.find(s2)
assert i1 < i2, f"E: '{s1}' not before '{s2}' in '{s}'"
return s[i1:i2]

====Binary, raw strings, html encoding====
# Binary Strings
sb = b"asdf"
# or
sb = str.encode("asdf")
s = sb.decode("ascii") # decode binary strings
sb = s.encode("ascii") # encode string to binary

# raw string
s = r"c:\Windows" # no escape of \ needed

# convert utf-8 to html umlaute
s = "Nürnberg".encode("ascii", "xmlcharrefreplace").decode()
# -> Nürnberg

====Guess encoding via chardet====
import chardet # pip install chardet
result = chardet.detect(raw_data)
if result["encoding"] and result["confidence"] > 0.5: # noqa: PLR2004
encoding = result["encoding"]
else:
encoding = "utf-8"

there is a much faster alternative [https://github.com/PyYoshi/cChardet cchardet], but that requires Microsoft Visual C++ 14.0

====Merge variables in string / sprintf====
print("Renner =", i)
print("Renner = %3d" % i) # leading 0's
print(f"Renner = {i}")

# place formatted numbers in a string / sprintf
s = "The {:.1f}% {} cost {:05.2f} euros".format( 5.1, "beer", 3.50)
print(s)
# > The 5.1% beer cost 03.50 euros

s = f"The length is {72.8958:.2f} meters"
# > The length is 72.90 meters

===Lists===
like @array in Perl
lst = [1, 2, 3, 4, 5, 6]
lst = [x / 2 for x in range(10)]
lst = [None] * 10 # Initiate list of None elements:
len(lst) # length
lst2 = lst[0:10] # get elements 0-10
for i in lst:
print(i)

====clone list====
# dangerous: creates a link to the original list
list_2 = my_list # M's elements are LINKS to L's
# clones can be achieved via:
list_2 = my_list.copy
list_2 = my_list[:]
list_2 = list(my_list)

====list modifications====
lst.pop() # returns and removes the last item
lst.pop(i) # returns and removes the item at position int i
lst.insert(i, x) # insert item x at position int i
lst.remove(x) # removes the first occurrence of item x
lst.append(x) # append a single element
lst.extend(m) # append elements of another list

lst.reverse() # reverse the order of the list
lst = sorted(["B", "a", "c"], key=str.casefold) # case insensitive / ignore case

# list to string
s = "".join(lst)
s = "\n".join(lst)
# string to list
lst = s.split() # default: split on space
lst = s.split(",")

# remove empty values from end
while L[-1] == "":
L.pop()

# check and remove items from list
# from https://stackoverflow.com/a/6024599
# iterates in-situ via reversed index
for i in range(len(lst) - 1, -1, -1):
element = lst[i]
if check(element):
del lst[i]

====check if item is in list====
x in lst
x not in lst
lst.count(x) # how many items x are in the list
i = lst.index("word") # find in list / returns the position of the first match in list

====pair 2 lists====
# zip: merge 2 lists to list of tuples
data = list(zip(data_x, data_y, strict=True))

# unzip: split list of pairs into 2 lists
data_x, data_y = zip(*data, strict=True)

# Cartesian product of lists / tuples
import itertools
for i in itertools.product(*list_of_lists):
print(i)

====sort multidim list====
lst = sorted(lst, key=lambda x: x[0], reverse=False)
lst = sorted(lst, key=lambda row: (row["Wann"], row ["Wer"]), reverse=False)

====filter list====
lines = [x for x in lines if not x.startswith ("word")]
lst = ["asdf", "asdf2", "qwertz"]
lst = [elem for elem in lst if "asdf" in elem]

====for each element====
# trim/strip spaces for each element
lines = [s.strip() for s in lines]
# modify each item in list by adding constant string
l = [s + ';' + v for v in l]

# modify item in list
for idx, line in enumerate(cont):
if "K1001/1" in line:
line = "K1001/1 Test Nr " + str(i) + "\n"
cont[idx] = line
break

===Tuples===
Ordered sequence, with no ability to replace or delete items
tpl = (1,2,3,4,5,6)

list -> tuple
tpl = tuple(lst)

combine 2 tuples
tpl = tpl1 + tpl2

===Dictionaries===
like %hash in Perl
d = {"key1": "value1", "key2": 123}
d["key3"] = (1, 2, 3)
del d["key3"]

len(d)
d.clear()
d.copy()
d.keys()
d.values()
d.items() # returns a list of tuples (key, value)
d.get(k) # returns value of key k
d.get(k, x) # returns value of key k; if k is not in d it returns x
count = d.get("key", 0) + 1 # nice for counters
d.pop(k) # returns and removes item k
d.pop(k, x) # returns and removes item k; if k is not in d it returns x
# checks
x in d
x not in d

# dict can have tuple as key
tpl = ("key1", "key2", 123)
d[tpl] = 123456

# loop over all key-value pairs
for key, value in d.items():
print(f"{key} = {value}")

# sorted keys:
for key in sorted(dict.keys()):
pass
# sorted values, reversed
for key, value in sorted(d.items(), key=lambda item: item[1], reverse=True):
pass

# join / merge 2 dicts
d.update(d2)

# flatten dict of dict
flat = d.copy()
meta = d.pop("metadata")
flat.update(meta)

====MultiDim Dictionaries====
d = {}
d["item1"] = {}
d["item1"]["value"] = 1.909e18
d["item1"]["name"] = "Item 1"
d["item2"] = {}
d["item2"]["value"] = 1.725e18
d["item2"]["name"] = "Item 2"

for k in d.keys():
d[k]["value"] = d[k]["value"] * 1.1

def find_common_strings(dict1: dict, dict2: dict) -> dict[Any, Any]:
"""Find common strings in two dict, returning a new dict of those strings."""

def recursive_match(d1: dict, d2: dict) -> dict[Any, Any]:
common_dict = {}
for key in d1: # noqa: PLC0206
if key in d2:
if isinstance(d1[key], dict) and isinstance(d2[key], dict):
# Recurse if both values are dicts
nested_common = recursive_match(d1[key], d2[key])
if nested_common: # Add to result if common values are found
common_dict[key] = nested_common
elif isinstance(d1[key], str) and isinstance(d2[key], str):
# Check if both are strings and identical
if d1[key] == d2[key]:
common_dict[key] = d1[key]
return common_dict

return recursive_match(dict1, dict2)

def update_dict_with_new_values(original_dict: dict, new_values: dict) -> dict:
"""Update strings in the original dict with new values from the new_values dict."""

def recursive_update(d1: dict, new_vals: dict) -> None:
for key in new_vals: # noqa: PLC0206
if key in d1:
if isinstance(d1[key], dict) and isinstance(new_vals[key], dict):
# Recurse if both values are dicts
recursive_update(d1[key], new_vals[key])
elif isinstance(d1[key], str) and isinstance(new_vals[key], str):
# Update the string
d1[key] = new_vals[key]

recursive_update(original_dict, new_values)
return original_dict

===Datetime, Date and Time===
import datetime as dt
from zoneinfo import ZoneInfo
TZ_DE = ZoneInfo("Europe/Berlin")
TZ_UTC = ZoneInfo("UTC")

====Create====
# date
date = dt.date(2023, 12, 31)
date = dt.date.fromisocalendar(int(year), int(week), int(daynum)) # daynum: 1..7
date = dt.date.fromisoformat("2020-03-10")
date = dt.datetime.strptime(datestr, "%y%m%d").date()
date_today = dt.datetime.now(tz=dt.UTC).date()
date_yesterday = dt.datetime.now(tz=TZ_DE).date() - dt.timedelta(days=1)
days_overdue = (date_completed - date_due).days
# to add one month (which is not supported by timedelta)
from dateutil.relativedelta import relativedelta
date_end = date_start + relativedelta(months=1)
delta_days = 1 + (end - start).days

# datetime
dt_now = dt.datetime.now(tz=TZ_DE)
my_dt = dt.datetime(2023, 12, 31, 14, 31, 56, tzinfo=TZ_DE) # 2023-12-31 14:31:56
my_dt = dt.datetime.fromtimestamp(my_timestamp, tz=TZ_DE)
my_dt = dt.datetime.fromisoformat ("2017-01-01T12:30:59.000000")
my_dt = dt.datetime.fromisoformat("2020-03-10 06:01:01+00:00")
s = "2020-03-10T06:01:01Z"
my_dt = dt.datetime.fromisoformat(s.replace("Z", " +00:00"))
my_date_local = my_dt.astimezone(tz=TZ_DE).date()

# datetime -> date
my_date = my_dt.date()
# date -> datetime
my_date = dt.date(2023, 12, 31)
my_dt = dt.datetime(my_date.year, my_date.month, my_date.day, tzinfo=TZ_DE)

# to string
date_str = my_dt.strftime("%y%m%d")
date_str = my_dt.strftime("%Y-%m-%d %H:%M:%S")
# alternative using time
date_str = time.strftime("%Y-%m-%d %H:%M:%S")
# now in UTC without milliseconds
date_str = dt.datetime.now(tz=dt.timezone.utc).replace(microsecond=0).isoformat() + "Z"

# remove timezone
my_dt = my_dt.replace(tzinfo=None)

# German format
import locale
locale.setlocale(locale.LC_ALL, "de_DE")
print(my_date.strftime("%a %x")) # Mo 25.12.2023

# Calendar week
week = my_date.isocalendar()[1]
print("KW%02d" % week)

# first day of quarter
def get_first_day_of_the_quarter(my_date: dt.date) -> dt.date:
return dt.date(my_date.year, 1 + 3 * ((my_date.month - 1) // 3), 1)

====rounding datetimes to minutes====
def floor_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Floor (=round down) minutes to X min resolution."""
min_new = res * (my_dt.minute // res)
return my_dt.replace(minute=min_new, second=0, microsecond=0)

def ceil_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Ceil (=round up) minutes to X min resolution."""
min_new = res * (1 + my_dt.minute // res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

def round_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Cound minutes to X min resolution."""
min_old_dec = float(my_dt.minute) + float(my_dt.second * 60)
min_new = res * round(min_old_dec / res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

my_dt = dt.datetime.fromisoformat("2011-11-04 00:05:23+00:00")
print(f"original: {my_dt}")
print(f"floored: {floor_dt_minutes(my_dt,5)}")
print(f"ceileded: {ceil_dt_minutes(my_dt,5)}")
print(f"rounded: {round_dt_minutes(my_dt,5)}")

====Timing / Time elapsed====
import time
timestart = time.time()
sec = time.time() - timestart
print(f"Time elapsed: {sec:.2f} sec")

==OS, Argpars, etc.==
===Hostname===
from socket import gethostname
print(gethostname())

===Checking Operating System===
import os
import sys

if os.name == "posix":
print("posix/Unix/Linux")
elif os.name == "nt":
print("windows")
else:
print("unknown os")
sys.exit(1) # throws exception, use quit() to close / die silently

===Get filename of python script===
my_file_path = __file__

alternative using sys package
from sys import argv
myFilename = argv[0]

===accessing os envrionment variables===
import os
print(os.getenv("tmp"))
# better:
try:
s = os.environ[key] # throws error if unset
except KeyError:
error_message(f"Environment variable {key} not set.")

===Command Line Arguments===
====ArgumentParser====
import argparse

parser = (
argparse.ArgumentParser()
) # construct the argument parser and parse the arguments
# -h comes automatically

# Boolean Parameter
parser.add_argument(
"-v", "--verbose", help="increase output verbosity", action="store_true"
) # store_true -> Boolean Value

# Choice Parameter
# restrict to a list of possible values / choices
# parser.add_argument("--choice", type=int, choices= [0, 1, 2], help="Test choices")

# Positional Parameter (like text.py 123)
# parser.add_argument("num", type=int, help="Number of things")

# Required Parameter
# parser.add_argument("-i", "--input", type=str, required=True, help="Path of file")

# Optional Parameter
parser.add_argument("-n", "--number", type=int, help="Number of clicks")
# Optional Parameter with Default
parser.add_argument(
"-s",
"--seconds",
type=int,
default=sec_default,
help="Duration of clicking, default = %i (sec)" % sec_default,
)

args = vars(parser.parse_args())

if args["verbose"]:
pass # do nothing
# print ("verbosity turned on")
if args["number"]:
print("num=%i" % args["number"])

==== match case statement ====
(new in python 3.10)
match args:
case {"sap": "prod", "version": 1}:
...
case {"sap": "prod", "version": 2}:
...
case _: # default case

==File Access==
===Pathlib===
for migrating see table [https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module]
from pathlib import Path

p = Path("dir/test.txt")

log_file = Path(__file__).with_suffix(".log") # __file__ is name of python script

my_fname = p.name # alternative to basename
my_ext = p.suffix
(filepath_without_ext, file_ext) = (p.stem, p.suffix)
my_dir = p.parent
my_parent_dir = p.parents[1]
p2 = p.with_suffix(".json")
p3 = p.parent / (p.name + "-autofix.tex")

# checks
Path(my_file_str).exists()
Path(my_file_str).is_file()
Path(my_file_str).is_dir()

# loop over glob of files matching wildcard
for file_out in Path("mydir").glob("*-autofix.tex"):

# loop over dirs
p = Path() # cwd
list_of_repos = [x for x in p.iterdir() if x.is_dir() and (x / ".git").is_dir()]

alternative using old os functions
Split path into folder, filename, ext
import os
(dirName, fileName) = os.path.split(f)
(fileBaseName, fileExtension) = os.path.splitext(fileName)
fileOut = os.path.splitext(fileIn)[0] + "-out.txt"

===File Manipulations: copy, move, delete, touch===
from pathlib import Path

# copy
from shutil import copyfile
copyfile(Path("file-source.txt"), Path("dir") / Path("file-target.txt"))

# move / rename
Path("file.txt").replace(Path("file2.txt"))

# delete
Path("file.txt").unlink()

# touch
Path("file.txt").touch()

===File size and timestamp===
# size
Path("file.txt").stat().st_size

# timestamp last modified
Path("file.txt").stat().st_mtime

===Dir create/mkdir and delete===
# create dir
from pathlib import Path
Path("myDir1/myDir2").mkdir(parents=True, exist_ok=True)

# delete dir
import shutil
shutil.rmtree(d)

===Loop over Directories===
====Fetch Dir Contents / Loop over Files====
glob via pathlib
for fileOut in Path("mydir").glob("*-autofix.tex"):

Get list of files in directory, filter dirs from list, filter by ext
dir= "/path/to/some/dir"
listoffiles = [ f for f in os.listdir(dir) if os.path.isfile(os.path.join(dir ,f)) and f.lower()[-4:] == ".gpx"]
listoffiles.sort()

====Traverse in Subdirs====
# walk into path an fetch all files matching extension jpe?g
files = []
for (dirpath, dirnames, filenames) in os.walk("."):
dirpath = dirpath.replace("\\", "/")
for file in filenames:
if file.endswith(".txt"):
files.append(dirpath + "/" + file)
elif re.search(r"\.jpe?g$", file, re.IGNORECASE):
files.append(dirpath + "/" + file)

==File Parsing==
===File General===
====File Read====
path_file_in = Path("file.txt")

# via pathlib
cont = path_file_in.read_text(encoding="utf-8") # note: utf-8-sig for UTF-8-BOM

# general
with path_file_in.open(encoding="utf-8") as fh:
cont = fh.read()
# or
lst = fh.readlines()
# or
line = fh.readline()
# or
for line in fh:
print(line)

fh = path_file_in.open(encoding="utf-8")
...
fh.close()

====File Write====
path_file_out = Path("out/1/out.txt")

# write via pathlib
path_file_out.write_text("some text")

# write general
with path_file_out.open(mode="w", encoding="utf-8", newline="\n") as fh:
# w = overWrite file ; a = append to file
# If running Python in Windows, "\n" is automatically replaced by "\r\n".
# To prevent this use newline='\n'
fh.writelines(lst) # no linebreaks
# or
fh.write("\n".join(lst))
# or
for line in lst:
fh.write(line)

# Force update of file contents without closing it
fh.flush()

# alternative
fh = path_file_out.open(mode="w", encoding="utf-8", newline="\n")
...
fh.close()

===CSV===
====CSV Read====
import csv
from pathlib import Path

with Path("data.csv").open(mode="r", encoding="utf-8") as fh: # for Excel use ANSI
csv_reader = csv.DictReader(fh, dialect="excel", delimiter="\t")
for row in csv_reader:
print(f'\t{row["name"]} works in the {row["department"]} department')

====CSV Write====
import csv
from pathlib import Path

# simple
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.writer(fh, delimiter="\t")
csvwriter.writerow(("Date", "Confirmed"))

# dictwriter
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.DictWriter(
fh,
delimiter="\t",
extrasaction="ignore",
fieldnames=["date", "occupied_percent", "occupied", "total"],
)
csvwriter.writeheader()
for d in my_list_of_dicts:
d["occupied_percent"] = round(100 * d["occupied"] / d["total"], 1)
csvwriter.writerow(d)

===JSON===
====JSON Read====
import json
# from file
with path_to_download_file.open(encoding="utf-8") as fh:
d_json = json.load(fh)
# from string
d_json = json.loads(response_text)

def json_read(file_path: Path) -> list[dict[str, str]]:
"""
Read JSON data from file.
"""
with file_path.open(encoding="utf-8") as fh:
json_data = json.load(fh)
return json_data

====JSON Write====
Write dict to file in JSON format, keeping utf-8 encoding
import json

with path_to_download_file.open(mode="w", encoding="utf-8", newline="\n") as fh:
json.dump(my_dict, fh, ensure_ascii=False, sort_keys=False, indent=2)

def json_write(file_path: Path, json_data: list[dict[str, str]]) -> None:
"""
Write JSON data to file.
"""
with file_path.open("w", encoding="utf-8", newline="\n") as fh:
json.dump(json_data, fh, ensure_ascii=False, sort_keys=False, indent=2)

===Excel===
====Excel Read====
import openpyxl
# or xlsxwriter for write-only

workbook = openpyxl.load_workbook(
pathToMyExcelFile,
data_only=True, # read values instead of formulas
read_only=True, # suppresses: "UserWarning: wmf image format is not supported so the image is being dropped"
)
sheet = workbook["mySheetName"]
# or fetch active sheet
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=34, column=1) # index start here with 1
print(cell.value)
# or
print(sheet.cell(column=col, row=row).value)

====Excel Write====
import openpyxl

workbook = openpyxl.Workbook()
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=i, column=j) # index starts at 1
cell.value = "asdf"
workbook.save("out.xlsx")

===Word===
====Word Read====
from io import BytesIO
from pathlib import Path
import pandas as pd
from docx import Document
from docx.document import Document as DocxDocument

def read_doc_to_df(p_doc: Path | BytesIO) -> pd.DataFrame: # noqa: C901, PLR0912
"""Read Word document to DataFrame."""
if isinstance(p_doc, Path):
doc = Document(str(p_doc))
elif isinstance(p_doc, BytesIO):
doc = Document(p_doc)

# list of multiple paragraphs per key
data: dict[str, dict[str, list[str]]] = {}

sections1: list[str] = []
section1 = ""
key = ""
section2 = ""
score = ""

for paragraph in doc.paragraphs:
assert paragraph.style is not None
text = paragraph.text.strip()
if paragraph.style.name == "Title": # doc title
continue
if paragraph.style.name == "Heading 1": # sheet name
section1 = text
if section1 not in sections1:
sections1.append(section1)
key = ""
section2 = ""
elif paragraph.style.name == "Heading 2": # Key of the question
key = text.split(" | ")[0]
key = section1 + "|" + key
section2 = ""
elif paragraph.style.name == "Normal" and text.startswith("[") and "]" in text:
section2, t2 = text[1:].split("]", maxsplit=1)
if section2 == "Score":
score = t2.strip()
data[key][section2] = [score]
elif paragraph.style.name == "Normal":
text = paragraph.text.strip()
# skip requirements leftovers and empty paragraphs
if (section2 == "" and text.startswith("...")) or text == "":
continue
lst = data.setdefault(key, {}).setdefault(section2, [])
lst.append(text)
data[key][section2] = lst
else:
msg = f"Unexpected style: {paragraph.style.name} in paragraph: {text}"
raise ValueError(msg)

# flatten the text from list to str
data2: dict[str, dict[str, str]] = {}
for key, sections in data.items():
for section2, lst in sections.items():
s = "\n".join(lst)
if key not in data2:
data2[key] = {}
data2[key][section2] = s.strip()

df1 = pd.DataFrame.from_dict(data2, orient="index")
df1.index.name = "Key"
df1 = df1.reset_index()
df1[ [ "Sheet", "Key" ] ] = df1["Key"].str.split("|", n=1, expand=True)

return df1

====Word Write====
from io import BytesIO
from pathlib import Path
import pandas as pd from docx import Document
from docx.document import Document as DocxDocument
from docx.shared import Cm

LAYOUT_SMALL_MARGINS = True
if LAYOUT_SMALL_MARGINS:
sections = doc.sections
for section in sections:
section.top_margin = Cm(1)
section.bottom_margin = Cm(1)
section.left_margin = Cm(1)
section.right_margin = Cm(1)

doc.add_heading("My Title", level=0)
doc.add_heading("My Section", level=1)
doc.add_paragraph("Some Text)
p = doc.add_paragraph()
runner = p.add_run(bold text")
runner.bold = True # runner.italic = True

==Regular Expressions==
See [http://docs.python.org/library/re.html]

See [https://pythex.org] for an online tester

multiple flags are joined via pipe |
s = re.sub("asdf.*", r"qwertz", s, flags=re.DOTALL | re.IGNORECASE)

====Lookahead and Lookbehind====
pos lookahead: (?=...)
neg lookahead: (?!...)
pos lookbehind (?<=...)
neg lookbehind (?<!...)

====matching====
import re

# V0: simple 1
myPattern = "(/\*\*\* 0097_210000_0192539580000_2898977_0050 \*\*\*/.*?)($|/\*\*\*)"
myRegExp = re.compile(myPattern, re.DOTALL)
myMatch = myRegExp.search(cont)
assert myMatch != None, f"golden file not found in file {filename}"
cont_golden = myMatch.group(1)

# V1: simple 2
assert (
re.match("^[a-z]{2}$", d_settings["country"]) != None
), f'Error: county must be 2 digit lower case. We got: {d_settings["country"]}'

# V2: find and count
was = r"""Part: ([^<+])"""
cnt_parts = len(re.findall(was, cont))
cont = re.sub(was, r"\n\nTeil: \1\n\n", cont)
assert cnt_parts == 4, f"{cnt_parts} == 4"
assert "Part:" not in cont

=====Match email=====
def checkValidEMail(email: str) -> bool:
# from https://stackoverflow.com/posts/719543/timeline bottom edit
if not re.fullmatch(r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$", email):
print("Error: invalid email")
quit()
return True

====Find all====
myMatches = re.findall('href="([^"]+)"', cont)
for myMatch in myMatches:
print(myMatch)

====substring====
import re

# simple via search
lk_id = re.search('^.*timeseries\-(\d+)\.json$', f).group(1)

# simple via sub
myPattern = "^.*" + s1 + "(.*)" + s2 + ".*$"
out = re.sub(myPattern, r"\1", s)

# more robust including an assert
def substr_between(s: str, s1: str, s2: str) -> str:
"""
returns substring of s, found between strings s1 and s2
s1 and s2 can be regular expressions
"""
myPattern = s1 + '(.*)' + s2
myRegExp = re.compile(myPattern)
myMatches = myRegExp.search(s)
assert myMatches != None, f"E: can't find '{s1}'...'{s2}' in '{s}'"
out = myMatches.group(1)
return out

matchObj = re.search(r"(\d+\.\d+)", text
if matchObj:
price = float( '%s' % (matchObj).group(0) )

====Naming of match groups====
(?P<name>...), see [https://docs.python.org/3/library/re.html]

====Search and Replace====
From [http://www.regular-expressions.info/python.html] 
re.sub(regex, replacement, str) performs a search-and-replace across subject, replacing all matches of regex in str with replacement. The result is returned by the sub() function. The str string you pass is not modified.

s = re.sub(" +", " ", s)

====Splitting====
From [http://docs.python.org/library/re.html] 
split() splits a string into a list delimited by the passed pattern. The method is invaluable for converting textual data into data structures that can be easily read and modified by Python as demonstrated in the following example that creates a phonebook.

First, here is the input. Normally it may come from a file, here we are using triple-quoted string syntax:
>>> input = """Ross McFluff: 834.345.1254 155 Elm Street
...
... Ronald Heathmore: 892.345.3428 436 Finley Avenue
... Frank Burger: 925.541.7625 662 South Dogwood Way
...
...
... Heather Albrecht: 548.326.4584 919 Park Place"""

The entries are separated by one or more newlines. Now we convert the string into a list with each nonempty line having its own entry:
>>> entries = re.split("\n+", input)
>>> entries
['Ross McFluff: 834.345.1254 155 Elm Street',
'Ronald Heathmore: 892.345.3428 436 Finley Avenue',
'Frank Burger: 925.541.7625 662 South Dogwood Way',
'Heather Albrecht: 548.326.4584 919 Park Place']

Finally, split each entry into a list with first name, last name, telephone number, and address. We use the maxsplit parameter of split() because the address has spaces, our splitting pattern, in it:
>>> [re.split(":? ", entry, 3) for entry in entries]
[['Ross', 'McFluff', '834.345.1254', '155 Elm Street'],
['Ronald', 'Heathmore', '892.345.3428', '436 Finley Avenue'],
['Frank', 'Burger', '925.541.7625', '662 South Dogwood Way'],
['Heather', 'Albrecht', '548.326.4584', '919 Park Place']]

==== replace cont by linebreaks====
def replace_cont_by_linebreaks(s: str, regex: str) -> str:
"""
Replace regex in s by the number of linebreaks it originally contained.
"""
myMatches = re.findall(regex, s, flags=re.DOTALL)
for match in myMatches:
linebreaks = match.count("\n")
s = s.replace(match, "\n" * linebreaks, 1)
return s

==Image/Picture/Photo==
from PIL import Image, ImageFilter # pip install Pillow

fileIn = "2018-02-09 13.56.25.jpg"
# Read image
img = Image.open(fileIn)
PROBLEM: 
PIL Image.save() drops the IPTC data like tags, keywords, copywrite, ... 
better using https://imagemagick.org instead when tags shall be kept

====Resize====
# Resize keeping aspect ration -> img.thumbnail
# drops exif data, exif can be added from source file via exif= in save, see below
size = 1920, 1920
img.thumbnail(size, Image.ANTIALIAS)

====Export file====
fileOut = os.path.splitext(fileIn)[0] + "-edit.jpg"
try:
img = Image.open(fileIn)
img.save(fp=fileOut, format="JPEG", quality='keep') # exif=dict_exif_bytes
# JPEG Parameters
# * qualitiy : 'keep' or 1 (worst) to 95 (best), default = 75. Values above 95 should be avoided.
# * dpi : tuple of integers representing the pixel density, (x,y)
except IOError:
print("cannot write file '%s'" % fileOut)

====Export Progressive / web optimized JPEG====
from PIL import ImageFile # for MAXBLOCK for progressive export
fileOut = os.path.splitext(fileIn)[0] + "-progressive.jpg"
try:
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)
except IOError:
ImageFile.MAXBLOCK = img.size[0] * img.size[1]
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)

====JPEG Meta Data: EXIF and IPTC====
=====IPTC: Tags/Keywords=====
from iptcinfo3 import IPTCInfo # this works in pyhton 3!
iptc = IPTCInfo(fileIn)
if len(iptc['keywords']) > 0: # or supplementalCategories or contacts
print('====> Keywords')
for key in sorted(iptc['keywords']):
s = key.decode('ascii') # decode binary strings
print(s)

=====EXIF via piexif=====
import piexif # pip install piexif
exif_dict = piexif.load(img.info['exif'])
print(exif_dict['GPS'][piexif.GPSIFD.GPSAltitude])
# returns list of 2 integers: value and donator -> v / d
# (340000, 1000) => 340m
# (51, 2) => 25.5m

# Modify altitude
exif_dict['GPS'][piexif.GPSIFD.GPSAltitude] = (140, 1) # 140m

# write to file
exif_bytes = piexif.dump(exif_dict)
fileOut = os.path.splitext(fileIn)[0] + "-modExif.jpg"
try:
img.save(fp=fileOut, format="jpeg", exif=exif_bytes, quality='keep')
except IOError:
print("cannot write file '%s'" % fileOut)
or
exif_dict = piexif.load(fileIn)
for ifd in ("0th", "Exif", "GPS", "1st"):
print("===" + ifd)
for tag in exif_dict[ifd]:
print(piexif.TAGS[ifd][tag]["name"], "\t",
tag, "\t", exif_dict[ifd][tag])
print(exif_dict['0th'][306]) # 306 = DateTime

=====EXIF via exifread=====
# Open image file for reading (binary mode)
fh = open(fileIn, "rb")
# Return Exif tags
exif = exifread.process_file(fh)
fh.close()
# for tag in exif.keys():
# if tag not in ('JPEGThumbnail', 'TIFFThumbnail', 'Filename', 'EXIF MakerNote'):
# print("%s\t%s" % (tag, exif[tag]))
print(exif["Image DateTime"])
print(exif["GPS GPSLatitude"])
print(exif["GPS GPSLongitude"])

=====EXIF GPS via PIL=====
# from https://developer.here.com/blog/getting-started-with-geocoding-exif-image-metadata-in-python3
def get_exif(filename):
image = Image.open(filename)
image.verify()
image.close()
return image._getexif()

def get_labeled_exif(exif):
labeled = {}
for (key, val) in exif.items():
labeled[TAGS.get(key)] = val
return labeled

def get_geotagging(exif):
if not exif:
raise ValueError("No EXIF metadata found")
geotagging = {}
for (idx, tag) in TAGS.items():
if tag == "GPSInfo":
if idx not in exif:
raise ValueError("No EXIF geotagging found")
for (key, val) in GPSTAGS.items():
if key in exif[idx]:
geotagging[val] = exif[idx][key]
return geotagging

def get_decimal_from_dms(dms, ref):
degrees = dms[0][0] / dms[0][1]
minutes = dms[1][0] / dms[1][1] / 60.0
seconds = dms[2][0] / dms[2][1] / 3600.0
if ref in ["S", "W"]:
degrees = -degrees
minutes = -minutes
seconds = -seconds
return round(degrees + minutes + seconds, 5)

def get_coordinates(geotags):
lat = get_decimal_from_dms(geotags["GPSLatitude"], geotags["GPSLatitudeRef"])
lon = get_decimal_from_dms(geotags["GPSLongitude"], geotags["GPSLongitudeRef"])
return (lat, lon)

exif = get_exif(fileIn)
exif_labeled = get_labeled_exif(exif)
print(exif_labeled["DateTime"])

geotags = get_geotagging(exif)
print(get_coordinates(geotags))

====Template Matching / Image Regocnition Using CV2 / OpenCV====
see [[Python - CV2]]

====Optical Character Recognition (OCR) ====
see [[Python - OCR]]

==Templates/Snippets==
===Retry on Exception===
for retry_no in range(MAX_RETRIES):
try:
if retry_no > 0:
logger.warning("retrying %d/%d...", retry_no + 1, MAX_RETRIES)
doit()
except json.JSONDecodeError:
logger.exception("JSONDecodeError")
if retry_no == MAX_RETRIES - 1:
# end of retries
raise

===Diff of 2 files===
import difflib

fh1 = p_file1.open("r", encoding="utf-8")
fh2 = p_file1.open("r", encoding="utf-8")
diff = difflib.ndiff(fh1.readlines(), fh2.readlines())
delta = "".join(line for line in diff if line.startswith(("+ ", "- ")))
print(delta)

===GPX parsing===
import gpxpy
import gpxpy.gpx
# Elevation data by NASA: see lib at https://github.com/tkrajina/srtm.py
fh_gpx_file = open(gpx_file_path, 'r')
gpx = gpxpy.parse(fh_gpx_file)
# Loops for accessing the data
for track in gpx.tracks:
for segment in track.segments:
for point in segment.points:
for waypoint in gpx.waypoints:
for route in gpx.routes:
for point in route.points:
# interesting properties of point / waypoint objects:
point.time
point.latitude
point.longitude
point.source
waypoint.name

===TypeHints===
from typing import Any, Dict, cast
creds = cast(dict[str, str], tomllib.load(f)) # type: ignore
o = cast(Dict[str, Any], tomllib.load(f)) # type: ignore
o["sap"] = cast(Dict[str, str], o["sap"])
o["settings"] = cast(Dict[str, str | int | bool], o["settings"])
o["settings"]["sleep_time"] = cast(int, o["settings"]["sleep_time"])

===Pylance/Pyright===
import tomllib
# shows warning: Import "xyz" could not be resolved
# fix by
import tomllib # pyright: ignore

===asserts function argument validation===
aus [https://bildungsportal.sachsen.de/opal/auth/RepositoryEntry/12518195218/CourseNode/94506982489539?0 Python Kurs von Carsten Knoll]
def eine_funktion(satz, ganzzahl, zahl2, liste):
if not type(satz) == str:
print "Datentpyfehler: satz"
return -1
if not isinstance(ganzzahl, int):
print "Datentpyfehler: ganzzahl"
return -2
if not isinstance(liste, (tuple, list)):
print "Datentpyfehler: liste"
return -3
# Kompakteste Variante (empfohlen):
assert zahl2 > 0, "Error: zahl2 ist nicht > 0" # Assertation-Error bei Nichterfuellung

def F(x):
if not isinstance(x, (float, int)):
msg = "Zahl erwartet, %s bekommen" % type(x)
raise ValueError(msg)
return x**2
better:
def F(x):
assert isinstance(x, (float, int)), "Error: x is not of type float or int"
return x**2

assert variant in [
"normal",
"gray",
"cannyedge",
], "Error: variant is not in 'normal', 'gray', 'cannyedge'"

===Exceptions===
see [https://medium.com/@saadjamilakhtar/5-best-practices-for-python-exception-handling-5e54b876a20]

====Catching====
Catch keyboard interrupt and do a "save exit"
try:
i = 0
while 1:
i += 1
print(i)
except KeyboardInterrupt:
print("stopped")

Catch '''all''' exceptions
try:
[...]
except Exception as e:
print("Exception: ", e)

====Raising Exceptions====
if resp.status_code != 200: # noqa: PLR2004
msg = f"Bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise ValueError(msg) from None

Custom Exceptions
try:
raise Exception("HiHo")

raise ValueError¶

===perl grep and map===
from [https://stackoverflow.com/questions/12845288/grep-on-elements-of-a-list]
def grep(list, pattern):
expr = re.compile(pattern)
return [elem for elem in list if expr.match(elem)]
or
filteredList = filter(lambda x: x < 7 and x > 2, unfilteredList)

def map(list, was, womit):
return list(map(lambda i: re.sub(was, womit, i), list))
# was = '.*"(\d+)".*'
# womit = r"\1"

===unit testing using pytest===
install via
pip install pytest
activate in vscode, see [https://code.visualstudio.com/docs/python/testing#_enable-a-test-framework]: To enable testing, use the Python: Configure Tests command on the Command Palette.

see https://docs.pytest.org/en/6.2.x/assert.html#assert

test_1.py:
import myLib # my custom lib to test

class TestClass:
def test_one(self):
x = "this"
assert "h" in x

def test_two(self):
assert myLib.multiply(1, 2) == 2

def test_three(self):
assert myLib.multiply(2, 2) == 2

====project structure====
=====simplest structore: test next to script=====
Alternative without tests dir:
./main.py
./helper.py
./helper_test.py
with helper_test.py:
from helper import fnc1

=====standalone model with src and tests dirs=====
for this dir structure
src/helper.py
tests/helper_test.py
helper_test.py needs
import sys
from pathlib import Path
sys.path.insert(0, (Path(__file__).parent.parent / "src").as_posix())

=====standalone model with tests dir=====
if the test files are placed inside a ./tests/ dir
./main.py
./helper.py
tests/helper_test.py
helper_test.py needs
sys.path.insert(0, Path(__file__).parent.parent.as_posix())
from helper import fnc1

====parametrize====
import pytest
@pytest.mark.parametrize(
("test_input", "expected"),
[("", None), ("PT30M", 30), ("PT3H", 3 * 60), ("PT2H30M", 2 * 60 + 30)],
)
def test_task_est_to_minutes(test_input: str, expected: int) -> None:
assert task_est_to_minutes(test_input) == expected

=====fixures: prepare and cleanup test_data=====
to automatically run before and after each testcase
@pytest.fixture(autouse=True)
def _setup_tests(): # noqa: ANN202
cache_prepare_lists()
cache_prepare_tasks()

yield

cache_cleanup_test_data()

====Test coverage report====
pip install pytest-cov
# console output
pytest --cov

# html report in coverage_report/index.html with details per file
pytest --cov --cov-report=html:coverage_report
to exclude a code block from coverage report, add
# pragma: no cover

=== Sleep / Wait for input ===
sleep for a while
import time
time.sleep(60)

wait for user input
input("press Enter to close")

===Suppress Warnings===
see [https://docs.python.org/3/library/warnings.html#temporarily-suppressing-warnings]
import warnings
warnings.filterwarnings("ignore", message=".*native_field_num.*not found in message.*")

===Logging===
====V4 minimal stdout logging====
import logging
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s")
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
logging.getLogger("httpx").setLevel(logging.WARNING)

# usage of variables and rounding of numbers. (%d also rounds floats)
LOGGER.info("file %s has %d lines with avg %.1f char/line", filename, lines, c_p_l)

====V3 simple logging====
# main file:
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(name)s\t%(message)s",
handlers=[
RotatingFileHandler(
Path(__file__).with_suffix(".log"),
maxBytes=10485760, # 10 MB = 10*1024*1024,
backupCount=1,
encoding="utf-8",
)
],
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(Path(__file__).stem)
logger.info("Start")

# other files:
import logging
logger = logging.getLogger(__name__)

====V2: rotating file and STDOUT====
# 1. setup
import logging
from logging.handlers import RotatingFileHandler

logfile = "myApp.log"
maxBytes = 20 * 1024 * 1024
backupCount = 5
loglevel_console = logging.INFO
loglevel_file = logging.DEBUG

# create logger
logger = logging.getLogger("root")
logger.setLevel(loglevel_file)

# console handler
ch = logging.StreamHandler()
ch.setLevel(loglevel_console)

# rotating file handler
# fh = logging.FileHandler(logfile)
fh = RotatingFileHandler(logfile, maxBytes=maxBytes, backupCount=backupCount)
fh.setLevel(loglevel_file)

# create formatter and add it to the handlers
# %(name)s = LoggerName, %(threadName)s = TreadName
formatter = logging.Formatter(
"%(asctime)s - %(levelname)s - %(name)s - %(threadName)s - %(message)s "
)
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(ch)
logger.addHandler(fh)

logger.debug("DebugMe")
logger.info("Starting")
logger.warning("Attention")
logger.error("Something went wrong")
logger.critical("Something seriously went wrong ")

# 2. in other files/modules now use
import logging

logger = logging.getLogger(__name__)
logger.info("text")

...
except Exception as e:
logger.exception("Unhandeled exception")
quit()

====V1: Simple====
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(message)s", # \t%(name)s
filename=Path(__file__).with_suffix(".log"), # uncomment to switch to stdout
filemode="a",
)
logger = logging.getLogger(__name__)
logger.debug("Some text")
logger.info("Some text")
logger.warning("Some text")
logger.error("Some text")
logger.critical("Some text")
try:
...
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise

===Process Bar===
see [https://tqdm.github.io tqdm]
from tqdm import tqdm
for i in tqdm(range(10000)):
....
or
with tqdm(total=total_iterations, desc="Processing") as progress_bar:
...
progress_bar.update(1)

===CGI Web development===
# Print necessary headers.
print("Content-Type: text/html")
print()

# errors and debugging info to browser
import cgitb
cgitb.enable()

Access URL or Form Parameters
# V2 from https://www.tutorialspoint.com/python/python_cgi_programming.htm
import cgi
form = cgi.FieldStorage()
username = form.getvalue('username')
print(username)

# V1
import sys
import urllib.parse
query = os.environ.get('QUERY_STRING')
query = urllib.parse.unquote(query, errors="surrogateescape")
d = dict(qc.split("=") for qc in query.split("&"))
print(d)

====CGI Backend Returning JSONs====
#!/usr/local/bin/python3.6
# -*- coding: utf-8 -*-

import cgi
import json

# Print necessary headers.
print("Content-type: application/json")
print()

def get_form_parameter(para: str) -> str:
"asserts that a given parameter is set and returns its value"
value = form.getvalue(para)
assert value, f"Error: parameter {para} missing"
assert value != "", f"Error: parameter {para} missing"
return value

response = {}
response['status'] = "ok"

try:
action = get_form_parameter("action")
response['action'] = action
if action == "myAction":
...

except Exception as e:
response['status'] = "error"
d = {"type": str(type(e)), "text": str(e)}
response["exception"] = d

finally:
print(json.dumps(response))

==Databases==
===PostgreSQL===
====Improved helper function====
"""Helper functions for DB access."""

import datetime as dt
import logging

import psycopg2
import psycopg2.extras
from creds_db import credentials as creds_db

# Configure logging
logger = logging.getLogger(__name__)

connection: psycopg2.extensions.connection = None # type: ignore
cursor_query: psycopg2.extensions.cursor = None # type: ignore
cursor_write: psycopg2.extensions.cursor = None # type: ignore

def connect() -> (
tuple[
psycopg2.extensions.connection,
psycopg2.extensions.cursor,
psycopg2.extensions.cursor,
]
):
"""Connect to the database."""
try:
connection = psycopg2.connect(**creds_db)
cursor_query = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)
cursor_write = connection.cursor()
logger.info(
"Connected to database %s on host %s",
creds_db["database"],
creds_db["host"],
)
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise
else:
return connection, cursor_query, cursor_write

def disconnect() -> None:
"""Disconnect from the database."""
try:
if cursor_query:
cursor_query.close()
if cursor_write:
cursor_write.close()
if connection:
connection.close()
logger.info("Disconnected from the database")
except psycopg2.DatabaseError:
logger.exception("Error during disconnection: %s")

def exists(url: str, valid_hours: float) -> bool | None:
"""
Check if a record exists.

Returns
-------
None for missing record
True for valid record
False for expired record

"""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=valid_hours)
sql = """
SELECT delete_at > %s AS "valid"
FROM my_table
WHERE url = %s LIMIT 1;
"""
cursor_query.execute(sql, (delete_at, url))
res = cursor_query.fetchone()
return res[0] if res else None

def query1(url: str) -> dict | None:
"""Execute a query."""
sql = """
SELECT *
FROM my_table
WHERE url = %s LIMIT 1
"""
cursor_query.execute(sql, (url,))
return cursor_query.fetchone() # type: ignore

def insert(url: str, response: str, delete_in_hours: float) -> None:
"""Insert a record."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=delete_in_hours)
sql = """
INSERT INTO my_table (url, response, delete_at)
VALUES (%s, %s, %s);
"""
cursor_write.execute(sql, (url, response, delete_at))
connection.commit()

def delete(url: str) -> None:
"""Delete a record."""
sql = """
DELETE FROM my_table
WHERE url = %s;
"""
cursor_write.execute(sql, (url,))
connection.commit()

def delete_expired(in_hours: float) -> None:
"""Delete records that expire in less than in_hours from now."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=in_hours)

sql = """
DELETE FROM my_table
WHERE delete_at < %s;
"""
cursor_write.execute(sql, (delete_at,))
connection.commit()

connection, cursor_query, cursor_write = connect()

====Basics====
import psycopg2
import psycopg2.extras

credentials = {
"host": "localhost",
"port": 5432,
"database": "myDB",
"user": "myUser",
"password": "myPwd",
}
connection = psycopg2.connect(**credentials)
cursor = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)

l_bind_vars = [["A1", "A2"], ["B1", "B2"]]

sql = """
SELECT * FROM myTable
WHERE 1=1
AND status NOT IN ('CLOSED')
AND ColA = %s
AND ColB = %s
ORDER BY created DESC
"""
cursor.execute(sql, l_bind_vars)
d_data = dict(cursor.fetchone())

====export result to csv file====
sql1 = "SELECT * FROM table"
sql2 = "COPY (" + sql1 + ") TO STDOUT WITH CSV HEADER DELIMITER '\t'"
with open("out.csv", "w") as file:
cursor.copy_expert(sql2, file)

====ReadConfigFile to connect to PostgreSQL====
from [http://www.postgresqltutorial.com/postgresql-python/connect/]
database.ini
[postgresql]
host=dbhost
port=5432
database=dbname
user=dbuser
password=dbpass

from configparser import ConfigParser

def config(filename="database.ini", section="postgresql"):
parser = ConfigParser()
parser.read(filename)
# get section, default to postgresql
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception(
"Section {0} not found in the {1} file".format(section, filename)
)
return db

main.py
import psycopg2
from config import config

def connect():
"""Connect to the PostgreSQL database server"""
conn = None
try:
params = config() # read connection parameters
print("Connecting to the PostgreSQL database...")
conn = psycopg2.connect(**params)
cur = conn.cursor() # create a cursor
print("PostgreSQL database version:")
cur.execute("SELECT version()") # execute a statement
db_version = cur.fetchone()
print(db_version)
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
print("Database connection closed.")

if __name__ == "__main__":
connect()

===SQL Lite / SQLite===
see page [[SQLite]]

===InfluxDB===
see [[InfluxDB]]

==Internet Access==
===Send E-Mails===
see [[Python - eMail]]

===Download data using browser UA / REST===
import requests

headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0",
}
resp = requests.get(
url,
headers=headers,
timeout=3, # timeout in sec, requests should always have a timeout!
# timeout=(1,3), # 1s connect-timout, 3s read-timeout
)
if resp.status_code != 200: # noqa: PLR2004
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download only if cache is too old====
import time
from pathlib import Path
import requests

def fetch_url_or_cache(file_cache: Path, url) -> str:
"""Fetch URL and cache in a file."""
if check_cache_file_available_and_recent(
file_path=file_cache, max_age=3600, verbose=False
):
with file_cache.open(mode="r", encoding="utf-8") as fh:
s = fh.read()
else:
s = fetch(url=url)
with file_cache.open(mode="w", encoding="utf-8", newline="\n") as fh:
fh.writelines(s)
return s

def check_cache_file_available_and_recent(
file_path: Path,
max_age: int = 3500,
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = False
if file_path.exists() and (time.time() - file_path.stat().st_mtime < max_age):
cache_good = True
return cache_good

# or
def check_cache_file_available_and_recent_verbose(
file_path: Path, max_age: int = 3600, *, verbose: bool = False
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = True
if not file_path.exists():
if verbose:
print(f"No Cache available: {file_path}")
cache_good = False
if cache_good and time.time() - (time.time() - file_path.stat().st_mtime < max_age):
if verbose:
print(f"Cache too old: {file_path}")
cache_good = False
return cache_good

def fetch(url) -> str:
"""Fetch URL."""
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0", # noqa: E501
}
resp = requests.get(url, headers=headers, timeout=5)
if resp.status_code == 200: # noqa: PLR2004
return resp.content.decode("ascii")
else: # noqa: RET505
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download using urllib.request====
from urllib.request import urlopen

url = "https://pomber.github.io/covid19/timeseries.json"
with urlopen(url) as response: # noqa: S310
response_text = response.read()

===HTML parsing and extracting of elements===
V2: via BeautifulSoup
from bs4 import BeautifulSoup # pip install beautifulsoup4

soup = BeautifulSoup(cont, features="html.parser")
my_element = soup.find("div", {"class": "user-formatted-inner"})
my_body = my_element.prettify()
# my_body = my_element.encode()
# my_body = str(my_element)

V1: via lxml and xpath
from lxml import html
import requests

page = requests.get(url)
tree = html.fromstring(page.content)
tbody_trs = tree.xpath("//*/tbody/tr")
l_rows = []
for tr in tbody_trs:
l_columns = []
if len(tr) != 15:
continue
for td in tr:
l_columns.append(td.text_content())
l_rows.append(list(l_columns))

===HTML entities to unicode===
import html
cont = html.unescape(cont)

==GUI Interactions==
===Take Screenshot===
import pyautogui # (c:\Python\Scripts\)pip install pyautogui
# pyautogui does only support screenshots on monitor #1
...
screenshot = pyautogui.screenshot()
# screenshot = pyautogui.screenshot(region=(screenshotX,screenshotY, screenshotW, screenshotH))
screenshot = np.array(screenshot)
# Convert RGB to BGR
screenshot = screenshot[:, :, ::-1].copy()

===Mouse Actions===
def clickIt(x,y,key="") :
x0, y0 = pyautogui.position()
if key != "": # crtl, shift
pyautogui.keyDown(key)
pyautogui.moveTo(x, y, duration=0.2)
pyautogui.click(x=x , y=y, button='left', clicks=1, interval=0.1)
if key != "": # crtl, shift
pyautogui.keyUp(key)
pyautogui.moveTo(x0, y0)

===Web Automation===
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time
import glob

class StravaUserMapDL():
def __init__(self):
self.driver = webdriver.Firefox()

def login(self):
driver = self.driver
url = "https://www.somewebpage.com"
email = "myemail"
password = "mypassword"
driver.get(url)

title = driver.title
urlIs = driver.current_url
cont = driver.page_source # as string
FILE = open(filename,"w") # w = overWrite file ; a = append to file
FILE.write(cont)
FILE.close()

# handle login if urlIs != url
if (urlIs != url):
# activate checkbox 'remember_me'
elem = driver.find_element_by_id('remember_me')
if (elem.is_selected() == False):
elem.click()
assert elem.is_selected() == True
elem = driver.find_element_by_id('email')
elem.send_keys(email)
elem = driver.find_element_by_id('password')
elem.send_keys(password)
elem.send_keys(Keys.RETURN)
# Wait until login pages is replaced by real page
urlIs = driver.current_url
while (urlIs != url):
time.sleep(1)
urlIs = driver.current_url
print (urlIs)

# results = driver.find_elements_by_class_name('following')
# results = driver.find_elements_by_tag_name('li')

# print(results[0].text)
assert (urlIs == url)

===Unit Tests using Web Automation===
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time

class PythonOrgSearch(unittest.TestCase):
# def __init__(self,asdf):
# self.driver = webdriver.Firefox()

def setUp(self):
print ("setUp")
# headless mode:
# opts = Options()
# opts.set_headless()
# assert opts.headless # Operating in headless mode
# self.driver = webdriver.Firefox(options=opts)

self.driver = webdriver.Firefox()

def test_search_in_python_org(self):
driver = self.driver
driver.get("http://www.python.org")
self.assertIn("Python", driver.title)
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
print ("fertig: python_org")

def tearDown(self):
print ("tearDown")
print ("close Firefox")
self.driver.close() # close tab
self.driver.quit() # quit browser
# os._exit(1) # exit unittest without Exception

if __name__ == "__main__":
try:
unittest.main()
except SystemExit as e:
os._exit(1)

==Cryptography and Hashing==
====Hashing via SHA256====
def gen_SHA256_string(s: str) -> str:
m = hashlib.sha256()
m.update(s.encode("ascii"))
return m.hexdigest()

====Hashing via MD5====
(MD5 is not secure, better use SHA256)
def gen_MD5_string(s: str) -> str:
m = hashlib.md5()
m.update(s.encode("ascii"))
return m.hexdigest()

====Password hashing via bcrypt====
import bcrypt
pwd = 'geheim'
pwd = pwd.encode("utf-8")
# or
pwd = b'geheim'

hashed = bcrypt.hashpw(pwd, bcrypt.gensalt())
if bcrypt.checkpw(pwd, hashed):
print("It Matches!")
print(hashed.decode("utf-8"))

To use version 2a instead of 2b (default):
bcrypt.gensalt(prefix=b"2a")

==Multiprocessing, subprocesses and Threading==
see [[Python_-_Multithreading]] as well

use processes for CPU limited work 
use threads for I/O limited work

===Simple single process===
import subprocess
process = subprocess.run(["sudo", "du", "--max-depth=1", mydir], capture_output=True, text=True)
print (process.stdout)
old, depricated way:
os.system( "gnuplot " + gpfile)

===Multiprocessing===
see [[Python - Multithreading]] as well

V2 using pool and starmap
import multiprocessing
import os

def worker(i: int, s: str) -> list:
result = (i, s, os.getpid())
return result

if __name__ == "__main__":
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n" + str(i))
l_pile_of_work.append((tup))
# gen pool of processes
num_processes = min(multiprocessing.cpu_count(), len(l_pile_of_work))
pool = multiprocessing.Pool(processes=num_processes)
# start processes on pile of work
l_results_unsorted = pool.starmap(
func=worker, iterable=l_pile_of_work # each item is a list of 2 parameters
)

# or if only one parameter:
# l_results_unsorted = pool.map(doit_de_district, l_pile_of_work)

l_results = sorted(l_results_unsorted) # sort by i

V1
import subprocess
l_subprocesses = [] # queue list of subprocesses
max_processes = 4

def process_enqueue(new_process_parameters):
global l_subprocesses
# wait for free slot
while len(l_subprocesses) >= max_processes:
process_remove_finished_from_queue()
time.sleep(0.1) # sleep 0.1s
process = subprocess.Popen(new_process_parameters,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
l_subprocesses.append(process)

def process_remove_finished_from_queue():
global l_subprocesses
i = 0
while i <= len(l_subprocesses) - 1:
process = l_subprocesses[i]
if process.poll != None: # has already finished
process_print_output(process)
l_subprocesses.pop(i)
else: # still running
i += 1

def process_print_output(process):
"""waits for process to finish and prints process output"""
stdout, stderr = process.communicate()
if stdout != '':
print(f'Out: {stdout}')
if stderr != '':
print(f'ERROR: {stderr}')

def process_wait_for_all_finished():
global l_subprocesses
for process in l_subprocesses:
process_print_output(process)
l_subprocesses = [] # empty list of done subprocesses

process_enqueue(l_parameters1)
...
process_enqueue(l_parameters999)
process_wait_for_all_finished()

===Threading===
import threading
import queue
import os
import time

def worker(q_work: queue.Queue, results: dict):
while not q_work.empty():
i, s = q_work.get()
time.sleep(.1)
result = (i, s, os.getpid())
results[i] = result
q_work.task_done()

if __name__ == '__main__':
d_results = {} # threads can write into dict
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n"+str(i))
l_pile_of_work.append((tup))
# convert list of work to queue
q_pile_of_work = queue.Queue(
maxsize=len(l_pile_of_work)) # maxsize=0 -> unlimited
for params in l_pile_of_work:
q_pile_of_work.put(params)
# gen threads
num_threads = 100
l_threads = [] # List of threads, not used here
for i in range(num_threads):
t = threading.Thread(name='myThread-'+str(i),
target=worker,
args=(q_pile_of_work, d_results),
daemon=True)
l_threads.append(t)
t.start()
q_pile_of_work.join() # wait for all threas to complete
l_results_unsorted = d_results.values()
l_results = sorted(l_results_unsorted) # sort by i

===asyncio — Asynchronous I/O===
see https://docs.python.org/3/library/asyncio-task.html
import asyncio
import time

# basics
# task = asyncio.create_task(coro())
# Wrap the coro coroutine into a Task and schedule its execution. Return the Task object.

# sleep
# await asyncio.sleep(1)

# Running Tasks Concurrently and gathers the return values in list L
# L = await asyncio.gather(coro(x1,y1), coro(x2,y2), coro(x3,y3))

async def say_after(delay, what):
# Coroutines declared with the async/await syntax
await asyncio.sleep(delay)
print(what)

async def main():
# The asyncio.create_task() function to run coroutines concurrently as asyncio Tasks.
task1 = asyncio.create_task(
say_after(1, 'hello'))

task2 = asyncio.create_task(
say_after(1, 'world'))

print(f"started at {time.strftime('%X')}")

# Wait until both tasks are completed
await task1
await task2

print(f"finished at {time.strftime('%X')}")

asyncio.run(main())

==Pandas==
see [[Pandas]]

==Matplotlib==
see [[Matplotlib]]

== GUI via tkinter==
import tkinter as tk # no need to install via pip

class App(tk.Tk):
def __init__(self):
super().__init__()
self.title("robot-CClicker")
self.geometry("200x200+0+1080")
self.resizable(width=False, height=False)

self.l_buttons = []
self.__create_widgets()

def __create_widgets(self):
self.btn_click500 = tk.Button(
master=self,
text="500 clicks",
width=20,
command=lambda: self.clickBigCookie(500),
)
self.l_buttons.append(self.btn_click500)

for button in self.l_buttons:
button.pack(anchor=tk.W)

def clickBigCookie(self, num):
# TODO: the disabling of the buttons is not working
for button in self.l_buttons:
button["state"] = tk.DISABLED
helper.clickIt(self.posBigCockie[0], self.posBigCockie[1], num=num)
for button in self.l_buttons:
button["state"] = tk.NORMAL

if __name__ == "__main__":
app = App()
app.mainloop()

==Protobuf==
===convert .proto file to python class===
see https://www.datascienceblog.net/post/programming/essential-protobuf-guide-python/
protoc my_message.proto --python_out ./

===read/decode protobuf message===
parse message from file
import machine_message_pb2

with open("out.bin", "rb") as f:
my_message = my_message_pb2.my_message()
my_message.ParseFromString(f.read())
print(machine_message)

===create/encode protobuf message===
import machine_message_pb2

my_message = my_message_pb2.my_message()
my_message.data.field1 = 1
my_message.data.field2 = "asdf"

with open("out.bin", "wb") as f:
f.write(my_message.data.SerializeToString())

==venv / virtual environments==
pip install --upgrade pip
python -m venv .venv --prompt $(basename $(pwd))
source .venv/bin/activate
pip install -r requirements.txt
...
deactivate

==Pydantic data validation==
===Function input parameter validation===
see https://docs.pydantic.dev/usage/validation_decorator/
@validate_arguments
def sum(x: int, y: float) -> float:
print(x, y)
return x + y

print(sum(1.1, 1.1))
# -> 2.1

==Read config file==
===TOML===
see https://learnxinyminutes.com/docs/toml/

minimal example:
config.toml
# SAP API endpoint
[general]
sleep_time = 60
use_proxy = true
tool.py
try:
import tomllib # comes with python3.11
except ModuleNotFoundError:
import tomli as tomllib # pip install tomli
with open("config.toml", "rb") as f:
o = tomllib.load(f) # type: ignore
if o["settings"]["use_proxy"]:
...
for validation, see https://realpython.com/python-toml/

====TOML Write====
In for deployments I sometime want to modify a TOML config file for production, based on the dev version.
import tomli_w # pip install tomli-w
p_in = Path(__file__).parent.parent / ".streamlit/config.toml"
p_out = p_in.parent / "config-prod.toml"
with p_in.open("rb") as fh:
o = tomllib.load(fh)
o["server"]["fileWatcherType"] = "none"
del o["server"]["address"]
with p_out.open("wb") as fh:
tomli_w.dump(o, fh)

===ConfigParser: INI Config File Reading===
better use TOML

Config.ini
[Section1]
Cursor = 205E18
Grandma = 18E18
Farm = 11E18
Mine = 514E18
Factory = 155E18
test.py
from configparser import ConfigParser

config = ConfigParser(
interpolation=None
) # interpolation=None -> treats % in values as char % instead of interpreting it
config.read("Config.ini", encoding="utf-8")

print(config.getint("Section1", "key1"))
print(config.getfloat("Section1", "key2"))
print(config.get("Section1", "key3"))

for sec in config.sections():
d_settings = {}
for key in config.options(sec):
value = config.get(sec, key)
d_settings[key] = value
print("%15s : %s" % (key, value))

==VCard / vcf==
import codecs
import vobject # pip install vobject

obj = vobject.readComponents(codecs.open(file_in, encoding="utf-8").read()) # type: ignore
contacts: list[vobject.base.Component] = list(obj) # type: ignore

card = contacts[0]

if "bday" not in card.contents:
continue

# bday: remove 'VALUE': ['DATE']
card.contents["bday"][0].params = {} # type: ignore

# remove all fields but "bday", "n"
for key in card.contents.copy(): # loop over copy, to allow for deleting keys
if key not in ("n", "bday"):
del card.contents[key]

# recreate fn based on n
card.add("fn")
n = card.contents["n"][0]
fn = f"{n.value.given} {n.value.additional} {n.value.family}" # type: ignore
fn = re.sub(r"\s+", " ", fn)
card.fn.value = fn # type: ignore

with open("out.vcf", mode="w", encoding="utf-8", newline="\n") as fhOut:
fhOut.write(card.serialize())

==Sentry Exception monitoring==
see [[Sentry]]
==MQTT==
#!/usr/bin/env python3.12
"""Read power data from Tasmota device via MQTT."""

import json
from typing import Any

import paho.mqtt.client as mqtt
from mqtt_credentials import hostname, password, port, username
from paho.mqtt.reasoncodes import ReasonCode

def on_connect(
mqtt_client: mqtt.Client,
userdata: Any,
flags: dict[str, int],
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client connects MQTT broker.""" # noqa: D401
if reason_code == 0:
print("Connected to MQTT")
# print(f"MQTT protocol version: {mqtt_client._protocol}")

# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
mqtt_client.message_callback_add("tele/tasmota_MT681/SENSOR", on_message)
mqtt_client.subscribe([("tele/tasmota_MT681/SENSOR", 0)])

else:
print(f"Connection to MQTT broker failed. Error: {reason_code}")

def on_disconnect(
mqtt_client: mqtt.Client,
userdata: Any,
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client is disconnected from the MQTT broker.""" # noqa: D401
if reason_code > 0:
print(f"Unexpected disconnection. Error: {reason_code}")

def on_message(
mqtt_client: mqtt.Client,
userdata: Any,
message: mqtt.MQTTMessage,
) -> None:
"""Callback when a Tasmota message is received from the MQTT broker.""" # noqa: D401
s = message.payload.decode()
data = json.loads(s)
# {'Time': '2024-03-16T06:44:08', 'MT681': {'Total_in': 16.4396, 'Power_cur': 80, 'Total_out': 14.6403}} # noqa: E501

total_i = data["MT681"]["Total_in"]
total_o = data["MT681"]["Total_out"]

print(f"In:\t{total_i}\nOut:\t{total_o}")

mqtt_client.disconnect()
mqtt_client.loop_stop()

try:
# Create an MQTT client instance
mqtt_client: mqtt.Client = mqtt.Client(
mqtt.CallbackAPIVersion.VERSION2, "Python-Client-1"
) # type: ignore
mqtt_client.username_pw_set(username, password)

mqtt_client.on_connect = on_connect

# Connect to the MQTT broker
mqtt_client.connect(hostname, port, keepalive=60)

# NOT: while True, as that eats the CPU !!!
mqtt_client.loop_forever()
except KeyboardInterrupt:
print("KeyboardInterrupt, disconnecting from MQTT broker.")
mqtt_client.disconnect()
mqtt_client.loop_stop()

==Caching Functions==
from functools import lru_cache

@lru_cache(maxsize=128) # None equals to @cache declarator
def fnc(n:int):

==Performance/Resources==
===RAM/Memory consumption/bottleneck===
import tracemalloc
tracemalloc.start()

# ...
# Code
# example:
lst = list(range(1_000_000))
# ...

max_bytes = tracemalloc.get_traced_memory()[0]
print(f"{round(max_bytes / 10**6)}MB")

tracemalloc.stop()

===Profiling / Count and Runtime per Function===
call_stats = {}

def track_function_usage(func: Callable) -> Callable:
"""Annotation for gathering runtime statistics."""

@wraps(func)
def wrapper(*args: tuple, **kwargs: dict): # noqa: ANN202
start_time = time.time()
result = func(*args, **kwargs) # Call the original function
end_time = time.time()
call_stats[func.__name__]["calls"] += 1
call_stats[func.__name__]["total_time"] += end_time - start_time
return result
return wrapper

==JWT Token==
try:
decoded_token = jwt.decode(token, options={"verify_signature": False})
except jwt.ExpiredSignatureError:
msg = "Token has expired."
logger.exception(msg)
except jwt.InvalidTokenError:
msg = "Invalid token."
logger.exception(msg)

==Streamlit==
see https://github.com/entorb/streamlit-examples for a collection of my examples

===Minimum Example===
# src/app.py
from pathlib import Path
import streamlit as st
from streamlit.logger import get_logger
logger = get_logger(Path(__file__).stem)
logger.info("Start")
st.set_page_config(page_title="AppTitle", page_icon=None, layout="wide")
st.title("MyPageTitle")
st.header("MyHeader")
st.subheader("MySubHeader")
sel_param = st.selectbox("Parameter", ("count", "sum"))

# .streamlit/config.toml
[browser]
gatherUsageStats = false

[server]
port = 8501

# run it
streamlit run src/app.py

====Reduce Logging for K8s Deployed Docker Containers====
# reduce log output
for logger_name in [
"streamlit.runtime.caching.cache_utils",
"streamlit.runtime.caching.storage.in_memory_cache_storage_wrapper",
"streamlit.runtime.runtime",
"urllib3.connectionpool",
]:
get_logger(logger_name).setLevel(logging.WARNING)

===Chart via Altair===
====Line Chart====
chart = st.line_chart(df["value"])
# corresponds to
import altair as alt
c = (
alt.Chart(df)
.mark_line()
.encode(
x=alt.X("created", title=None),
y=alt.Y("value", title=None),
)
)
st.altair_chart(c, use_container_width=True)

====Bar Chart====
chart = (
alt.Chart(df)
.mark_bar()
.encode(
x=alt.X("yearmonth(date):T", title="Month"),
y=alt.Y(f"count:Q", title=None),
color="status:N",
tooltip=["yearmonth(date):T", "status:N", "cnt:Q"],
)
# .properties(width=600, height=400)
)
st.altair_chart(chart, use_container_width=True)

:T stands for Temporal. It indicates that the field contains date or time values.
:N stands for Nominal. It indicates that the field contains categorical data, which represents discrete categories or labels.
:Q stands for Quantitative. It indicates that the field contains numerical data that can be measured and compared.

===Excel Download===
def excel_download_buttons(df: pd.DataFrame, file_name: str = "export.xlsx") -> None:
"""Show prepare data and download buttons."""
if st.button(label="Click to prepare download"):
buffer = io.BytesIO()
with pd.ExcelWriter(buffer, engine="xlsxwriter") as writer:
df.to_excel(writer, sheet_name="Sheet1", index=False)
writer.close()

st.download_button(
label="Download data as Excel",
data=buffer,
file_name=file_name,
mime="application/vnd.ms-excel",
)

==UV Package Manager==
# create/update lockfile
uv lock --upgrade
# sync venv
uv sync --upgrade
# extract package info to requirements.txt
uv export --format requirements.txt --no-dev --no-hashes -o requirements.txt

Update Python version for my current project
uv venv --python=3.13.7

===Update UV===
uv self update
# or
brew update && brew upgrade uv

===Update Python Versions===
uv python upgrade

===UV + Docker on readOnlyRootFilesystem===
Example 1: https://github.com/SWE-Group-23/CM22007---Source/blob/10d02004f3a4208f7b30d37a0a87e89532a23575/create-service.sh#L117

Dockerfile
WORKDIR /app
COPY . .

ENV CASS_DRIVER_NO_EXTENSIONS=1

RUN --mount=type=cache,target=/root/.cache/uv \\
--mount=type=bind,source=uv.lock,target=uv.lock \\
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \\
uv sync --frozen --no-install-project

RUN uv pip install -e shared/

RUN addgroup -S group1 && adduser -S user1 -G group1 -u 1000
RUN chown -R user1:group1 /app
USER user1:group1

CMD ["uv", "run", "main.py"]

Deployment
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /home/user1/.cache/uv
name: uv
- mountPath: /tmp
name: temp

Example 2: [https://hynek.me/articles/docker-uv/ Production-ready Python Docker Containers with uv]

Dockerfile (without the comments)
ENV UV_LINK_MODE=copy \
UV_COMPILE_BYTECODE=1 \
UV_PYTHON_DOWNLOADS=never \
UV_PYTHON=python3.12 \
UV_PROJECT_ENVIRONMENT=/app

RUN --mount=type=cache,target=/root/.cache \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync \
--locked \
--no-dev \
--no-install-project

COPY . /src
WORKDIR /src
RUN --mount=type=cache,target=/root/.cache \
uv sync \
--locked \
--no-dev \
--no-editable

== Vulture: Search for dead code ==
see [https://github.com/jendrikseipp/vulture GitHub]

pyproject.toml
[tool.vulture]
paths = ["src"]
# exclude = ["src/models.py"]
ignore_names = [
"LOGGER",
]

.pre-commit-config.yaml
# Vulture: find dead code
- repo: https://github.com/jendrikseipp/vulture
rev: v2.16
hooks:
- id: vulture
pass_filenames: false

run
pip install vulture
vulture src/

uv add --dev vulture
uv run vulture src/

Python

2026-04-12T12:50:51Z

Torben: /* Vulture: Search for dead code */

[[Category:Coding]][[Category:Python]]

==Getting Started==
===Install===
====Python====
Best use [https://docs.astral.sh/uv/ UV] for managing python and package versions.

* for Windows: get and install Python from https://www.python.org
* for MacOS: use [https://github.com/pyenv/pyenv?tab=readme-ov-file#a-getting-pyenv pyenv]
brew install xz
brew install pyenv

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[ [ -d $PYENV_ROOT/bin ] ] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init - zsh)"' >> ~/.zshrc
exec "$SHELL"

pyenv install --list
pyenv install 3.13.5
pyenv global 3.13.5

vim ~/.zshrc
# add
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi

====Editor: Visual Studio Code====
excellent and free source-code editor that supports many languages.
* get and install from https://code.visualstudio.com

See Wickie page [[Visual_Studio_Code]] for general setup, extension, config...

Python Extensions
* Python
* Pylance
* [https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff Ruff Formater and Linter]
* [https://marketplace.visualstudio.com/items/?itemName=tamasfe.even-better-toml Even Better TOML]
Settings (CTRL + ,)
* Extensions -> Python -> Formatting: Provider = black
* Text Editor -> Editor: Format On Save
* Text Editor -> Files:Eol -> \n
* Linter: Ruff
Settings in settings.json
"python.analysis.completeFunctionParens": true,
"python.analysis.autoImportCompletions": true,
"python.analysis.inlayHints.functionReturnTypes": true,
"python.analysis.typeCheckingMode": "strict",
// Ruff
"[python]": {
"editor.defaultFormatter": "charliermarsh.ruff",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.fixAll": "explicit",
"source.organizeImports.ruff": "explicit"
},
},

Run your code
CTRL + F5 : run
F5 : run in debugger

====Code Formatter Ruff====
use software for handling the code formatting like "ruff" or "black", where Ruff is much faster and additionally provides linting.
pip install ruff
than activate in editor like vs code

===My Template===
see latest version at https://github.com/entorb/template-python

"""Main file."""
# by Torben Menke https://entorb.net
import logging
from pathlib import Path

def init_logging() -> None:
"""Initialize logging."""
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(
level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s"
)
logging.getLogger("httpx").setLevel(logging.WARNING)

init_logging()
LOGGER = logging.getLogger(__name__)

def main():
LOGGER.info("Moin Moin")

if __name__ == "__main__":
main()

"""Other file"""
import logging
LOGGER = logging.getLogger(__name__)

===Compile to .exe===
pip install pyinstaller
pyinstaller --onefile --console your.py
# for Excel and Matplotlib these options are required
--hidden-import=openpyxl --hidden-import=matplotlib --hidden-import pandas.plotting._matplotlib
([[Python - py2exe]] is deprecated)

==Basics==
=== Naming Conventions===
[https://google.github.io/styleguide/pyguide.html Google Python Style Guide]:
module_name, package_name, ClassName, method_name, ExceptionName, function_name, GLOBAL_CONSTANT_NAME, global_var_name, instance_var_name, function_parameter_name, local_var_name

====Doc Strings====
It is best practice to start a file with a docstring:
"""My Script"""
This can be accessed linke this:
title = __doc__[:-1] # type: ignore

===Installing packages===
python -m pip install --upgrade pip

pip install somemodule
# or
pip3 install somemodule
# or read from file
pip install -r requirements.txt

# uninstall
pip uninstall somemodule

# using a web proxy
# set proxy for windows cmd session
SET HTTPS_PROXY=http://myProxy:8080
(afterwards --proxy setting below no longer required
or
pip install --proxy http://myProxy:8080 somemodule

# list outdated packages
pip list --outdated

# update package
pip install --upgrade pyinstaller

# updating all via Windows Powershell (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | %{$_.split('==')[0]} | %{pip install --upgrade $_}

# updating all via Bash (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install --upgrade

# downgrade
pip install --upgrade pandas==1.2.4

====Oneline Updater for requirements.txt====
https://pkgui.com/pip

====update packages====
pip install pip-review
pip-review --auto

====generate requirements.txt====
pip install pipreqs
pipreqs ./

===Loops===
ATTENTION: The loops do not create a new variable scope. Only functions and modules introduce a new scope!
for p in all_files:
print(p)

for i, p in enumerate(all_files):
print(i, p)

for i in range(10):
print(i)
# del i

while i <= 100:
i += 1
...
if sth1:
continue # start next loop
if sth2:
break # exit loop

# inline if (requires a dummy else):
print("something") if sth else 0

===Math===
see [[Python - Math]] for linear regression

Modulo
15 % 4
# > 3

Integer Division
17 // 4
# > 4

====Random====
import random
random.randint(1000000, 9999999)

==Basic Objects==
===Variables===
del var # delete / undef a variable
var = None # sets to null

# check if variable is defined
if "var" in locals():
pass
# for object oriented projects:
if "var" in self.__dict__:
pass

# Access global variables in functions
var = 123

def test() -> None:
global var # point to global instead of creation of local var
var = 321

===Strings===
# num <-> str
s = str(i) # int to string
f = float(s) # str -> float
i = int(s)
str(round(f, 1)) # round first
# tests
s.isdigit() # 0-9
# note isdecimal() does also not match '1.1'

# printf: 1 digit
s = f"{value:0.1f}"
s = "%0.1f" % value

====Modify Strings====
# get string from prompt
s = input("Enter Text: ")

s = s.strip() # trim spaces from both sides, rstrip for right only
s = s.lower() # lower case
s = s.upper() # upper case
s = s.title() # upper case for first char of word

# upper case first letter of each word and also removes multiple and trailing spaces
import string
s = string.capwords(s)

# replace
s.replace(x, y)

# trim whitespaces from left and right
s.strip()

# replace all (multiple) whitespaces by single space ' '
s = " ".join(s.split())

# generate key value pairs from dict
# key1=value1&key2=value2
param_str = "&".join("=".join(tup) for tup in dict.items())

# repeat string multiple times
s * 5 # = s+s+s+s+s

====Substrings====
# find a substring:
if x in s: # True / False
pass
if len(s) > 0:
pass

# handling substrings
a = "abcd"
b = a[:1] + "o" + a[2:]
# > 'aocd'

s = "Hello there !bob@"
i1 = s.find("!") + 1
i2 = s.find("@")
substr = s[i1:i2]

def substr_between(s: str, s1: str, s2: str) -> str:
assert s1 in s, f"E: can't find '{s1}' in '{s}'"
assert s2 in s, f"E: can't find '{s1}' in '{s}'"
i1 = s.find(s1) + len(s1)
i2 = s.find(s2)
assert i1 < i2, f"E: '{s1}' not before '{s2}' in '{s}'"
return s[i1:i2]

====Binary, raw strings, html encoding====
# Binary Strings
sb = b"asdf"
# or
sb = str.encode("asdf")
s = sb.decode("ascii") # decode binary strings
sb = s.encode("ascii") # encode string to binary

# raw string
s = r"c:\Windows" # no escape of \ needed

# convert utf-8 to html umlaute
s = "Nürnberg".encode("ascii", "xmlcharrefreplace").decode()
# -> Nürnberg

====Guess encoding via chardet====
import chardet # pip install chardet
result = chardet.detect(raw_data)
if result["encoding"] and result["confidence"] > 0.5: # noqa: PLR2004
encoding = result["encoding"]
else:
encoding = "utf-8"

there is a much faster alternative [https://github.com/PyYoshi/cChardet cchardet], but that requires Microsoft Visual C++ 14.0

====Merge variables in string / sprintf====
print("Renner =", i)
print("Renner = %3d" % i) # leading 0's
print(f"Renner = {i}")

# place formatted numbers in a string / sprintf
s = "The {:.1f}% {} cost {:05.2f} euros".format( 5.1, "beer", 3.50)
print(s)
# > The 5.1% beer cost 03.50 euros

s = f"The length is {72.8958:.2f} meters"
# > The length is 72.90 meters

===Lists===
like @array in Perl
lst = [1, 2, 3, 4, 5, 6]
lst = [x / 2 for x in range(10)]
lst = [None] * 10 # Initiate list of None elements:
len(lst) # length
lst2 = lst[0:10] # get elements 0-10
for i in lst:
print(i)

====clone list====
# dangerous: creates a link to the original list
list_2 = my_list # M's elements are LINKS to L's
# clones can be achieved via:
list_2 = my_list.copy
list_2 = my_list[:]
list_2 = list(my_list)

====list modifications====
lst.pop() # returns and removes the last item
lst.pop(i) # returns and removes the item at position int i
lst.insert(i, x) # insert item x at position int i
lst.remove(x) # removes the first occurrence of item x
lst.append(x) # append a single element
lst.extend(m) # append elements of another list

lst.reverse() # reverse the order of the list
lst = sorted(["B", "a", "c"], key=str.casefold) # case insensitive / ignore case

# list to string
s = "".join(lst)
s = "\n".join(lst)
# string to list
lst = s.split() # default: split on space
lst = s.split(",")

# remove empty values from end
while L[-1] == "":
L.pop()

# check and remove items from list
# from https://stackoverflow.com/a/6024599
# iterates in-situ via reversed index
for i in range(len(lst) - 1, -1, -1):
element = lst[i]
if check(element):
del lst[i]

====check if item is in list====
x in lst
x not in lst
lst.count(x) # how many items x are in the list
i = lst.index("word") # find in list / returns the position of the first match in list

====pair 2 lists====
# zip: merge 2 lists to list of tuples
data = list(zip(data_x, data_y, strict=True))

# unzip: split list of pairs into 2 lists
data_x, data_y = zip(*data, strict=True)

# Cartesian product of lists / tuples
import itertools
for i in itertools.product(*list_of_lists):
print(i)

====sort multidim list====
lst = sorted(lst, key=lambda x: x[0], reverse=False)
lst = sorted(lst, key=lambda row: (row["Wann"], row ["Wer"]), reverse=False)

====filter list====
lines = [x for x in lines if not x.startswith ("word")]
lst = ["asdf", "asdf2", "qwertz"]
lst = [elem for elem in lst if "asdf" in elem]

====for each element====
# trim/strip spaces for each element
lines = [s.strip() for s in lines]
# modify each item in list by adding constant string
l = [s + ';' + v for v in l]

# modify item in list
for idx, line in enumerate(cont):
if "K1001/1" in line:
line = "K1001/1 Test Nr " + str(i) + "\n"
cont[idx] = line
break

===Tuples===
Ordered sequence, with no ability to replace or delete items
tpl = (1,2,3,4,5,6)

list -> tuple
tpl = tuple(lst)

combine 2 tuples
tpl = tpl1 + tpl2

===Dictionaries===
like %hash in Perl
d = {"key1": "value1", "key2": 123}
d["key3"] = (1, 2, 3)
del d["key3"]

len(d)
d.clear()
d.copy()
d.keys()
d.values()
d.items() # returns a list of tuples (key, value)
d.get(k) # returns value of key k
d.get(k, x) # returns value of key k; if k is not in d it returns x
count = d.get("key", 0) + 1 # nice for counters
d.pop(k) # returns and removes item k
d.pop(k, x) # returns and removes item k; if k is not in d it returns x
# checks
x in d
x not in d

# dict can have tuple as key
tpl = ("key1", "key2", 123)
d[tpl] = 123456

# loop over all key-value pairs
for key, value in d.items():
print(f"{key} = {value}")

# sorted keys:
for key in sorted(dict.keys()):
pass
# sorted values, reversed
for key, value in sorted(d.items(), key=lambda item: item[1], reverse=True):
pass

# join / merge 2 dicts
d.update(d2)

# flatten dict of dict
flat = d.copy()
meta = d.pop("metadata")
flat.update(meta)

====MultiDim Dictionaries====
d = {}
d["item1"] = {}
d["item1"]["value"] = 1.909e18
d["item1"]["name"] = "Item 1"
d["item2"] = {}
d["item2"]["value"] = 1.725e18
d["item2"]["name"] = "Item 2"

for k in d.keys():
d[k]["value"] = d[k]["value"] * 1.1

def find_common_strings(dict1: dict, dict2: dict) -> dict[Any, Any]:
"""Find common strings in two dict, returning a new dict of those strings."""

def recursive_match(d1: dict, d2: dict) -> dict[Any, Any]:
common_dict = {}
for key in d1: # noqa: PLC0206
if key in d2:
if isinstance(d1[key], dict) and isinstance(d2[key], dict):
# Recurse if both values are dicts
nested_common = recursive_match(d1[key], d2[key])
if nested_common: # Add to result if common values are found
common_dict[key] = nested_common
elif isinstance(d1[key], str) and isinstance(d2[key], str):
# Check if both are strings and identical
if d1[key] == d2[key]:
common_dict[key] = d1[key]
return common_dict

return recursive_match(dict1, dict2)

def update_dict_with_new_values(original_dict: dict, new_values: dict) -> dict:
"""Update strings in the original dict with new values from the new_values dict."""

def recursive_update(d1: dict, new_vals: dict) -> None:
for key in new_vals: # noqa: PLC0206
if key in d1:
if isinstance(d1[key], dict) and isinstance(new_vals[key], dict):
# Recurse if both values are dicts
recursive_update(d1[key], new_vals[key])
elif isinstance(d1[key], str) and isinstance(new_vals[key], str):
# Update the string
d1[key] = new_vals[key]

recursive_update(original_dict, new_values)
return original_dict

===Datetime, Date and Time===
import datetime as dt
from zoneinfo import ZoneInfo
TZ_DE = ZoneInfo("Europe/Berlin")
TZ_UTC = ZoneInfo("UTC")

====Create====
# date
date = dt.date(2023, 12, 31)
date = dt.date.fromisocalendar(int(year), int(week), int(daynum)) # daynum: 1..7
date = dt.date.fromisoformat("2020-03-10")
date = dt.datetime.strptime(datestr, "%y%m%d").date()
date_today = dt.datetime.now(tz=dt.UTC).date()
date_yesterday = dt.datetime.now(tz=TZ_DE).date() - dt.timedelta(days=1)
days_overdue = (date_completed - date_due).days
# to add one month (which is not supported by timedelta)
from dateutil.relativedelta import relativedelta
date_end = date_start + relativedelta(months=1)
delta_days = 1 + (end - start).days

# datetime
dt_now = dt.datetime.now(tz=TZ_DE)
my_dt = dt.datetime(2023, 12, 31, 14, 31, 56, tzinfo=TZ_DE) # 2023-12-31 14:31:56
my_dt = dt.datetime.fromtimestamp(my_timestamp, tz=TZ_DE)
my_dt = dt.datetime.fromisoformat ("2017-01-01T12:30:59.000000")
my_dt = dt.datetime.fromisoformat("2020-03-10 06:01:01+00:00")
s = "2020-03-10T06:01:01Z"
my_dt = dt.datetime.fromisoformat(s.replace("Z", " +00:00"))
my_date_local = my_dt.astimezone(tz=TZ_DE).date()

# datetime -> date
my_date = my_dt.date()
# date -> datetime
my_date = dt.date(2023, 12, 31)
my_dt = dt.datetime(my_date.year, my_date.month, my_date.day, tzinfo=TZ_DE)

# to string
date_str = my_dt.strftime("%y%m%d")
date_str = my_dt.strftime("%Y-%m-%d %H:%M:%S")
# alternative using time
date_str = time.strftime("%Y-%m-%d %H:%M:%S")
# now in UTC without milliseconds
date_str = dt.datetime.now(tz=dt.timezone.utc).replace(microsecond=0).isoformat() + "Z"

# remove timezone
my_dt = my_dt.replace(tzinfo=None)

# German format
import locale
locale.setlocale(locale.LC_ALL, "de_DE")
print(my_date.strftime("%a %x")) # Mo 25.12.2023

# Calendar week
week = my_date.isocalendar()[1]
print("KW%02d" % week)

# first day of quarter
def get_first_day_of_the_quarter(my_date: dt.date) -> dt.date:
return dt.date(my_date.year, 1 + 3 * ((my_date.month - 1) // 3), 1)

====rounding datetimes to minutes====
def floor_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Floor (=round down) minutes to X min resolution."""
min_new = res * (my_dt.minute // res)
return my_dt.replace(minute=min_new, second=0, microsecond=0)

def ceil_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Ceil (=round up) minutes to X min resolution."""
min_new = res * (1 + my_dt.minute // res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

def round_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Cound minutes to X min resolution."""
min_old_dec = float(my_dt.minute) + float(my_dt.second * 60)
min_new = res * round(min_old_dec / res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

my_dt = dt.datetime.fromisoformat("2011-11-04 00:05:23+00:00")
print(f"original: {my_dt}")
print(f"floored: {floor_dt_minutes(my_dt,5)}")
print(f"ceileded: {ceil_dt_minutes(my_dt,5)}")
print(f"rounded: {round_dt_minutes(my_dt,5)}")

====Timing / Time elapsed====
import time
timestart = time.time()
sec = time.time() - timestart
print(f"Time elapsed: {sec:.2f} sec")

==OS, Argpars, etc.==
===Hostname===
from socket import gethostname
print(gethostname())

===Checking Operating System===
import os
import sys

if os.name == "posix":
print("posix/Unix/Linux")
elif os.name == "nt":
print("windows")
else:
print("unknown os")
sys.exit(1) # throws exception, use quit() to close / die silently

===Get filename of python script===
my_file_path = __file__

alternative using sys package
from sys import argv
myFilename = argv[0]

===accessing os envrionment variables===
import os
print(os.getenv("tmp"))
# better:
try:
s = os.environ[key] # throws error if unset
except KeyError:
error_message(f"Environment variable {key} not set.")

===Command Line Arguments===
====ArgumentParser====
import argparse

parser = (
argparse.ArgumentParser()
) # construct the argument parser and parse the arguments
# -h comes automatically

# Boolean Parameter
parser.add_argument(
"-v", "--verbose", help="increase output verbosity", action="store_true"
) # store_true -> Boolean Value

# Choice Parameter
# restrict to a list of possible values / choices
# parser.add_argument("--choice", type=int, choices= [0, 1, 2], help="Test choices")

# Positional Parameter (like text.py 123)
# parser.add_argument("num", type=int, help="Number of things")

# Required Parameter
# parser.add_argument("-i", "--input", type=str, required=True, help="Path of file")

# Optional Parameter
parser.add_argument("-n", "--number", type=int, help="Number of clicks")
# Optional Parameter with Default
parser.add_argument(
"-s",
"--seconds",
type=int,
default=sec_default,
help="Duration of clicking, default = %i (sec)" % sec_default,
)

args = vars(parser.parse_args())

if args["verbose"]:
pass # do nothing
# print ("verbosity turned on")
if args["number"]:
print("num=%i" % args["number"])

==== match case statement ====
(new in python 3.10)
match args:
case {"sap": "prod", "version": 1}:
...
case {"sap": "prod", "version": 2}:
...
case _: # default case

==File Access==
===Pathlib===
for migrating see table [https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module]
from pathlib import Path

p = Path("dir/test.txt")

log_file = Path(__file__).with_suffix(".log") # __file__ is name of python script

my_fname = p.name # alternative to basename
my_ext = p.suffix
(filepath_without_ext, file_ext) = (p.stem, p.suffix)
my_dir = p.parent
my_parent_dir = p.parents[1]
p2 = p.with_suffix(".json")
p3 = p.parent / (p.name + "-autofix.tex")

# checks
Path(my_file_str).exists()
Path(my_file_str).is_file()
Path(my_file_str).is_dir()

# loop over glob of files matching wildcard
for file_out in Path("mydir").glob("*-autofix.tex"):

# loop over dirs
p = Path() # cwd
list_of_repos = [x for x in p.iterdir() if x.is_dir() and (x / ".git").is_dir()]

alternative using old os functions
Split path into folder, filename, ext
import os
(dirName, fileName) = os.path.split(f)
(fileBaseName, fileExtension) = os.path.splitext(fileName)
fileOut = os.path.splitext(fileIn)[0] + "-out.txt"

===File Manipulations: copy, move, delete, touch===
from pathlib import Path

# copy
from shutil import copyfile
copyfile(Path("file-source.txt"), Path("dir") / Path("file-target.txt"))

# move / rename
Path("file.txt").replace(Path("file2.txt"))

# delete
Path("file.txt").unlink()

# touch
Path("file.txt").touch()

===File size and timestamp===
# size
Path("file.txt").stat().st_size

# timestamp last modified
Path("file.txt").stat().st_mtime

===Dir create/mkdir and delete===
# create dir
from pathlib import Path
Path("myDir1/myDir2").mkdir(parents=True, exist_ok=True)

# delete dir
import shutil
shutil.rmtree(d)

===Loop over Directories===
====Fetch Dir Contents / Loop over Files====
glob via pathlib
for fileOut in Path("mydir").glob("*-autofix.tex"):

Get list of files in directory, filter dirs from list, filter by ext
dir= "/path/to/some/dir"
listoffiles = [ f for f in os.listdir(dir) if os.path.isfile(os.path.join(dir ,f)) and f.lower()[-4:] == ".gpx"]
listoffiles.sort()

====Traverse in Subdirs====
# walk into path an fetch all files matching extension jpe?g
files = []
for (dirpath, dirnames, filenames) in os.walk("."):
dirpath = dirpath.replace("\\", "/")
for file in filenames:
if file.endswith(".txt"):
files.append(dirpath + "/" + file)
elif re.search(r"\.jpe?g$", file, re.IGNORECASE):
files.append(dirpath + "/" + file)

==File Parsing==
===File General===
====File Read====
path_file_in = Path("file.txt")

# via pathlib
cont = path_file_in.read_text(encoding="utf-8") # note: utf-8-sig for UTF-8-BOM

# general
with path_file_in.open(encoding="utf-8") as fh:
cont = fh.read()
# or
lst = fh.readlines()
# or
line = fh.readline()
# or
for line in fh:
print(line)

fh = path_file_in.open(encoding="utf-8")
...
fh.close()

====File Write====
path_file_out = Path("out/1/out.txt")

# write via pathlib
path_file_out.write_text("some text")

# write general
with path_file_out.open(mode="w", encoding="utf-8", newline="\n") as fh:
# w = overWrite file ; a = append to file
# If running Python in Windows, "\n" is automatically replaced by "\r\n".
# To prevent this use newline='\n'
fh.writelines(lst) # no linebreaks
# or
fh.write("\n".join(lst))
# or
for line in lst:
fh.write(line)

# Force update of file contents without closing it
fh.flush()

# alternative
fh = path_file_out.open(mode="w", encoding="utf-8", newline="\n")
...
fh.close()

===CSV===
====CSV Read====
import csv
from pathlib import Path

with Path("data.csv").open(mode="r", encoding="utf-8") as fh: # for Excel use ANSI
csv_reader = csv.DictReader(fh, dialect="excel", delimiter="\t")
for row in csv_reader:
print(f'\t{row["name"]} works in the {row["department"]} department')

====CSV Write====
import csv
from pathlib import Path

# simple
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.writer(fh, delimiter="\t")
csvwriter.writerow(("Date", "Confirmed"))

# dictwriter
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.DictWriter(
fh,
delimiter="\t",
extrasaction="ignore",
fieldnames=["date", "occupied_percent", "occupied", "total"],
)
csvwriter.writeheader()
for d in my_list_of_dicts:
d["occupied_percent"] = round(100 * d["occupied"] / d["total"], 1)
csvwriter.writerow(d)

===JSON===
====JSON Read====
import json
# from file
with path_to_download_file.open(encoding="utf-8") as fh:
d_json = json.load(fh)
# from string
d_json = json.loads(response_text)

def json_read(file_path: Path) -> list[dict[str, str]]:
"""
Read JSON data from file.
"""
with file_path.open(encoding="utf-8") as fh:
json_data = json.load(fh)
return json_data

====JSON Write====
Write dict to file in JSON format, keeping utf-8 encoding
import json

with path_to_download_file.open(mode="w", encoding="utf-8", newline="\n") as fh:
json.dump(my_dict, fh, ensure_ascii=False, sort_keys=False, indent=2)

def json_write(file_path: Path, json_data: list[dict[str, str]]) -> None:
"""
Write JSON data to file.
"""
with file_path.open("w", encoding="utf-8", newline="\n") as fh:
json.dump(json_data, fh, ensure_ascii=False, sort_keys=False, indent=2)

===Excel===
====Excel Read====
import openpyxl
# or xlsxwriter for write-only

workbook = openpyxl.load_workbook(
pathToMyExcelFile,
data_only=True, # read values instead of formulas
read_only=True, # suppresses: "UserWarning: wmf image format is not supported so the image is being dropped"
)
sheet = workbook["mySheetName"]
# or fetch active sheet
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=34, column=1) # index start here with 1
print(cell.value)
# or
print(sheet.cell(column=col, row=row).value)

====Excel Write====
import openpyxl

workbook = openpyxl.Workbook()
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=i, column=j) # index starts at 1
cell.value = "asdf"
workbook.save("out.xlsx")

===Word===
====Word Read====
from io import BytesIO
from pathlib import Path
import pandas as pd
from docx import Document
from docx.document import Document as DocxDocument

def read_doc_to_df(p_doc: Path | BytesIO) -> pd.DataFrame: # noqa: C901, PLR0912
"""Read Word document to DataFrame."""
if isinstance(p_doc, Path):
doc = Document(str(p_doc))
elif isinstance(p_doc, BytesIO):
doc = Document(p_doc)

# list of multiple paragraphs per key
data: dict[str, dict[str, list[str]]] = {}

sections1: list[str] = []
section1 = ""
key = ""
section2 = ""
score = ""

for paragraph in doc.paragraphs:
assert paragraph.style is not None
text = paragraph.text.strip()
if paragraph.style.name == "Title": # doc title
continue
if paragraph.style.name == "Heading 1": # sheet name
section1 = text
if section1 not in sections1:
sections1.append(section1)
key = ""
section2 = ""
elif paragraph.style.name == "Heading 2": # Key of the question
key = text.split(" | ")[0]
key = section1 + "|" + key
section2 = ""
elif paragraph.style.name == "Normal" and text.startswith("[") and "]" in text:
section2, t2 = text[1:].split("]", maxsplit=1)
if section2 == "Score":
score = t2.strip()
data[key][section2] = [score]
elif paragraph.style.name == "Normal":
text = paragraph.text.strip()
# skip requirements leftovers and empty paragraphs
if (section2 == "" and text.startswith("...")) or text == "":
continue
lst = data.setdefault(key, {}).setdefault(section2, [])
lst.append(text)
data[key][section2] = lst
else:
msg = f"Unexpected style: {paragraph.style.name} in paragraph: {text}"
raise ValueError(msg)

# flatten the text from list to str
data2: dict[str, dict[str, str]] = {}
for key, sections in data.items():
for section2, lst in sections.items():
s = "\n".join(lst)
if key not in data2:
data2[key] = {}
data2[key][section2] = s.strip()

df1 = pd.DataFrame.from_dict(data2, orient="index")
df1.index.name = "Key"
df1 = df1.reset_index()
df1[ [ "Sheet", "Key" ] ] = df1["Key"].str.split("|", n=1, expand=True)

return df1

====Word Write====
from io import BytesIO
from pathlib import Path
import pandas as pd from docx import Document
from docx.document import Document as DocxDocument
from docx.shared import Cm

LAYOUT_SMALL_MARGINS = True
if LAYOUT_SMALL_MARGINS:
sections = doc.sections
for section in sections:
section.top_margin = Cm(1)
section.bottom_margin = Cm(1)
section.left_margin = Cm(1)
section.right_margin = Cm(1)

doc.add_heading("My Title", level=0)
doc.add_heading("My Section", level=1)
doc.add_paragraph("Some Text)
p = doc.add_paragraph()
runner = p.add_run(bold text")
runner.bold = True # runner.italic = True

==Regular Expressions==
See [http://docs.python.org/library/re.html]

See [https://pythex.org] for an online tester

multiple flags are joined via pipe |
s = re.sub("asdf.*", r"qwertz", s, flags=re.DOTALL | re.IGNORECASE)

====Lookahead and Lookbehind====
pos lookahead: (?=...)
neg lookahead: (?!...)
pos lookbehind (?<=...)
neg lookbehind (?<!...)

====matching====
import re

# V0: simple 1
myPattern = "(/\*\*\* 0097_210000_0192539580000_2898977_0050 \*\*\*/.*?)($|/\*\*\*)"
myRegExp = re.compile(myPattern, re.DOTALL)
myMatch = myRegExp.search(cont)
assert myMatch != None, f"golden file not found in file {filename}"
cont_golden = myMatch.group(1)

# V1: simple 2
assert (
re.match("^[a-z]{2}$", d_settings["country"]) != None
), f'Error: county must be 2 digit lower case. We got: {d_settings["country"]}'

# V2: find and count
was = r"""Part: ([^<+])"""
cnt_parts = len(re.findall(was, cont))
cont = re.sub(was, r"\n\nTeil: \1\n\n", cont)
assert cnt_parts == 4, f"{cnt_parts} == 4"
assert "Part:" not in cont

=====Match email=====
def checkValidEMail(email: str) -> bool:
# from https://stackoverflow.com/posts/719543/timeline bottom edit
if not re.fullmatch(r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$", email):
print("Error: invalid email")
quit()
return True

====Find all====
myMatches = re.findall('href="([^"]+)"', cont)
for myMatch in myMatches:
print(myMatch)

====substring====
import re

# simple via search
lk_id = re.search('^.*timeseries\-(\d+)\.json$', f).group(1)

# simple via sub
myPattern = "^.*" + s1 + "(.*)" + s2 + ".*$"
out = re.sub(myPattern, r"\1", s)

# more robust including an assert
def substr_between(s: str, s1: str, s2: str) -> str:
"""
returns substring of s, found between strings s1 and s2
s1 and s2 can be regular expressions
"""
myPattern = s1 + '(.*)' + s2
myRegExp = re.compile(myPattern)
myMatches = myRegExp.search(s)
assert myMatches != None, f"E: can't find '{s1}'...'{s2}' in '{s}'"
out = myMatches.group(1)
return out

matchObj = re.search(r"(\d+\.\d+)", text
if matchObj:
price = float( '%s' % (matchObj).group(0) )

====Naming of match groups====
(?P<name>...), see [https://docs.python.org/3/library/re.html]

====Search and Replace====
From [http://www.regular-expressions.info/python.html] 
re.sub(regex, replacement, str) performs a search-and-replace across subject, replacing all matches of regex in str with replacement. The result is returned by the sub() function. The str string you pass is not modified.

s = re.sub(" +", " ", s)

====Splitting====
From [http://docs.python.org/library/re.html] 
split() splits a string into a list delimited by the passed pattern. The method is invaluable for converting textual data into data structures that can be easily read and modified by Python as demonstrated in the following example that creates a phonebook.

First, here is the input. Normally it may come from a file, here we are using triple-quoted string syntax:
>>> input = """Ross McFluff: 834.345.1254 155 Elm Street
...
... Ronald Heathmore: 892.345.3428 436 Finley Avenue
... Frank Burger: 925.541.7625 662 South Dogwood Way
...
...
... Heather Albrecht: 548.326.4584 919 Park Place"""

The entries are separated by one or more newlines. Now we convert the string into a list with each nonempty line having its own entry:
>>> entries = re.split("\n+", input)
>>> entries
['Ross McFluff: 834.345.1254 155 Elm Street',
'Ronald Heathmore: 892.345.3428 436 Finley Avenue',
'Frank Burger: 925.541.7625 662 South Dogwood Way',
'Heather Albrecht: 548.326.4584 919 Park Place']

Finally, split each entry into a list with first name, last name, telephone number, and address. We use the maxsplit parameter of split() because the address has spaces, our splitting pattern, in it:
>>> [re.split(":? ", entry, 3) for entry in entries]
[['Ross', 'McFluff', '834.345.1254', '155 Elm Street'],
['Ronald', 'Heathmore', '892.345.3428', '436 Finley Avenue'],
['Frank', 'Burger', '925.541.7625', '662 South Dogwood Way'],
['Heather', 'Albrecht', '548.326.4584', '919 Park Place']]

==== replace cont by linebreaks====
def replace_cont_by_linebreaks(s: str, regex: str) -> str:
"""
Replace regex in s by the number of linebreaks it originally contained.
"""
myMatches = re.findall(regex, s, flags=re.DOTALL)
for match in myMatches:
linebreaks = match.count("\n")
s = s.replace(match, "\n" * linebreaks, 1)
return s

==Image/Picture/Photo==
from PIL import Image, ImageFilter # pip install Pillow

fileIn = "2018-02-09 13.56.25.jpg"
# Read image
img = Image.open(fileIn)
PROBLEM: 
PIL Image.save() drops the IPTC data like tags, keywords, copywrite, ... 
better using https://imagemagick.org instead when tags shall be kept

====Resize====
# Resize keeping aspect ration -> img.thumbnail
# drops exif data, exif can be added from source file via exif= in save, see below
size = 1920, 1920
img.thumbnail(size, Image.ANTIALIAS)

====Export file====
fileOut = os.path.splitext(fileIn)[0] + "-edit.jpg"
try:
img = Image.open(fileIn)
img.save(fp=fileOut, format="JPEG", quality='keep') # exif=dict_exif_bytes
# JPEG Parameters
# * qualitiy : 'keep' or 1 (worst) to 95 (best), default = 75. Values above 95 should be avoided.
# * dpi : tuple of integers representing the pixel density, (x,y)
except IOError:
print("cannot write file '%s'" % fileOut)

====Export Progressive / web optimized JPEG====
from PIL import ImageFile # for MAXBLOCK for progressive export
fileOut = os.path.splitext(fileIn)[0] + "-progressive.jpg"
try:
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)
except IOError:
ImageFile.MAXBLOCK = img.size[0] * img.size[1]
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)

====JPEG Meta Data: EXIF and IPTC====
=====IPTC: Tags/Keywords=====
from iptcinfo3 import IPTCInfo # this works in pyhton 3!
iptc = IPTCInfo(fileIn)
if len(iptc['keywords']) > 0: # or supplementalCategories or contacts
print('====> Keywords')
for key in sorted(iptc['keywords']):
s = key.decode('ascii') # decode binary strings
print(s)

=====EXIF via piexif=====
import piexif # pip install piexif
exif_dict = piexif.load(img.info['exif'])
print(exif_dict['GPS'][piexif.GPSIFD.GPSAltitude])
# returns list of 2 integers: value and donator -> v / d
# (340000, 1000) => 340m
# (51, 2) => 25.5m

# Modify altitude
exif_dict['GPS'][piexif.GPSIFD.GPSAltitude] = (140, 1) # 140m

# write to file
exif_bytes = piexif.dump(exif_dict)
fileOut = os.path.splitext(fileIn)[0] + "-modExif.jpg"
try:
img.save(fp=fileOut, format="jpeg", exif=exif_bytes, quality='keep')
except IOError:
print("cannot write file '%s'" % fileOut)
or
exif_dict = piexif.load(fileIn)
for ifd in ("0th", "Exif", "GPS", "1st"):
print("===" + ifd)
for tag in exif_dict[ifd]:
print(piexif.TAGS[ifd][tag]["name"], "\t",
tag, "\t", exif_dict[ifd][tag])
print(exif_dict['0th'][306]) # 306 = DateTime

=====EXIF via exifread=====
# Open image file for reading (binary mode)
fh = open(fileIn, "rb")
# Return Exif tags
exif = exifread.process_file(fh)
fh.close()
# for tag in exif.keys():
# if tag not in ('JPEGThumbnail', 'TIFFThumbnail', 'Filename', 'EXIF MakerNote'):
# print("%s\t%s" % (tag, exif[tag]))
print(exif["Image DateTime"])
print(exif["GPS GPSLatitude"])
print(exif["GPS GPSLongitude"])

=====EXIF GPS via PIL=====
# from https://developer.here.com/blog/getting-started-with-geocoding-exif-image-metadata-in-python3
def get_exif(filename):
image = Image.open(filename)
image.verify()
image.close()
return image._getexif()

def get_labeled_exif(exif):
labeled = {}
for (key, val) in exif.items():
labeled[TAGS.get(key)] = val
return labeled

def get_geotagging(exif):
if not exif:
raise ValueError("No EXIF metadata found")
geotagging = {}
for (idx, tag) in TAGS.items():
if tag == "GPSInfo":
if idx not in exif:
raise ValueError("No EXIF geotagging found")
for (key, val) in GPSTAGS.items():
if key in exif[idx]:
geotagging[val] = exif[idx][key]
return geotagging

def get_decimal_from_dms(dms, ref):
degrees = dms[0][0] / dms[0][1]
minutes = dms[1][0] / dms[1][1] / 60.0
seconds = dms[2][0] / dms[2][1] / 3600.0
if ref in ["S", "W"]:
degrees = -degrees
minutes = -minutes
seconds = -seconds
return round(degrees + minutes + seconds, 5)

def get_coordinates(geotags):
lat = get_decimal_from_dms(geotags["GPSLatitude"], geotags["GPSLatitudeRef"])
lon = get_decimal_from_dms(geotags["GPSLongitude"], geotags["GPSLongitudeRef"])
return (lat, lon)

exif = get_exif(fileIn)
exif_labeled = get_labeled_exif(exif)
print(exif_labeled["DateTime"])

geotags = get_geotagging(exif)
print(get_coordinates(geotags))

====Template Matching / Image Regocnition Using CV2 / OpenCV====
see [[Python - CV2]]

====Optical Character Recognition (OCR) ====
see [[Python - OCR]]

==Templates/Snippets==
===Retry on Exception===
for retry_no in range(MAX_RETRIES):
try:
if retry_no > 0:
logger.warning("retrying %d/%d...", retry_no + 1, MAX_RETRIES)
doit()
except json.JSONDecodeError:
logger.exception("JSONDecodeError")
if retry_no == MAX_RETRIES - 1:
# end of retries
raise

===Diff of 2 files===
import difflib

fh1 = p_file1.open("r", encoding="utf-8")
fh2 = p_file1.open("r", encoding="utf-8")
diff = difflib.ndiff(fh1.readlines(), fh2.readlines())
delta = "".join(line for line in diff if line.startswith(("+ ", "- ")))
print(delta)

===GPX parsing===
import gpxpy
import gpxpy.gpx
# Elevation data by NASA: see lib at https://github.com/tkrajina/srtm.py
fh_gpx_file = open(gpx_file_path, 'r')
gpx = gpxpy.parse(fh_gpx_file)
# Loops for accessing the data
for track in gpx.tracks:
for segment in track.segments:
for point in segment.points:
for waypoint in gpx.waypoints:
for route in gpx.routes:
for point in route.points:
# interesting properties of point / waypoint objects:
point.time
point.latitude
point.longitude
point.source
waypoint.name

===TypeHints===
from typing import Any, Dict, cast
creds = cast(dict[str, str], tomllib.load(f)) # type: ignore
o = cast(Dict[str, Any], tomllib.load(f)) # type: ignore
o["sap"] = cast(Dict[str, str], o["sap"])
o["settings"] = cast(Dict[str, str | int | bool], o["settings"])
o["settings"]["sleep_time"] = cast(int, o["settings"]["sleep_time"])

===Pylance/Pyright===
import tomllib
# shows warning: Import "xyz" could not be resolved
# fix by
import tomllib # pyright: ignore

===asserts function argument validation===
aus [https://bildungsportal.sachsen.de/opal/auth/RepositoryEntry/12518195218/CourseNode/94506982489539?0 Python Kurs von Carsten Knoll]
def eine_funktion(satz, ganzzahl, zahl2, liste):
if not type(satz) == str:
print "Datentpyfehler: satz"
return -1
if not isinstance(ganzzahl, int):
print "Datentpyfehler: ganzzahl"
return -2
if not isinstance(liste, (tuple, list)):
print "Datentpyfehler: liste"
return -3
# Kompakteste Variante (empfohlen):
assert zahl2 > 0, "Error: zahl2 ist nicht > 0" # Assertation-Error bei Nichterfuellung

def F(x):
if not isinstance(x, (float, int)):
msg = "Zahl erwartet, %s bekommen" % type(x)
raise ValueError(msg)
return x**2
better:
def F(x):
assert isinstance(x, (float, int)), "Error: x is not of type float or int"
return x**2

assert variant in [
"normal",
"gray",
"cannyedge",
], "Error: variant is not in 'normal', 'gray', 'cannyedge'"

===Exceptions===
see [https://medium.com/@saadjamilakhtar/5-best-practices-for-python-exception-handling-5e54b876a20]

====Catching====
Catch keyboard interrupt and do a "save exit"
try:
i = 0
while 1:
i += 1
print(i)
except KeyboardInterrupt:
print("stopped")

Catch '''all''' exceptions
try:
[...]
except Exception as e:
print("Exception: ", e)

====Raising Exceptions====
if resp.status_code != 200: # noqa: PLR2004
msg = f"Bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise ValueError(msg) from None

Custom Exceptions
try:
raise Exception("HiHo")

raise ValueError¶

===perl grep and map===
from [https://stackoverflow.com/questions/12845288/grep-on-elements-of-a-list]
def grep(list, pattern):
expr = re.compile(pattern)
return [elem for elem in list if expr.match(elem)]
or
filteredList = filter(lambda x: x < 7 and x > 2, unfilteredList)

def map(list, was, womit):
return list(map(lambda i: re.sub(was, womit, i), list))
# was = '.*"(\d+)".*'
# womit = r"\1"

===unit testing using pytest===
install via
pip install pytest
activate in vscode, see [https://code.visualstudio.com/docs/python/testing#_enable-a-test-framework]: To enable testing, use the Python: Configure Tests command on the Command Palette.

see https://docs.pytest.org/en/6.2.x/assert.html#assert

test_1.py:
import myLib # my custom lib to test

class TestClass:
def test_one(self):
x = "this"
assert "h" in x

def test_two(self):
assert myLib.multiply(1, 2) == 2

def test_three(self):
assert myLib.multiply(2, 2) == 2

====project structure====
=====simplest structore: test next to script=====
Alternative without tests dir:
./main.py
./helper.py
./helper_test.py
with helper_test.py:
from helper import fnc1

=====standalone model with src and tests dirs=====
for this dir structure
src/helper.py
tests/helper_test.py
helper_test.py needs
import sys
from pathlib import Path
sys.path.insert(0, (Path(__file__).parent.parent / "src").as_posix())

=====standalone model with tests dir=====
if the test files are placed inside a ./tests/ dir
./main.py
./helper.py
tests/helper_test.py
helper_test.py needs
sys.path.insert(0, Path(__file__).parent.parent.as_posix())
from helper import fnc1

====parametrize====
import pytest
@pytest.mark.parametrize(
("test_input", "expected"),
[("", None), ("PT30M", 30), ("PT3H", 3 * 60), ("PT2H30M", 2 * 60 + 30)],
)
def test_task_est_to_minutes(test_input: str, expected: int) -> None:
assert task_est_to_minutes(test_input) == expected

=====fixures: prepare and cleanup test_data=====
to automatically run before and after each testcase
@pytest.fixture(autouse=True)
def _setup_tests(): # noqa: ANN202
cache_prepare_lists()
cache_prepare_tasks()

yield

cache_cleanup_test_data()

====Test coverage report====
pip install pytest-cov
# console output
pytest --cov

# html report in coverage_report/index.html with details per file
pytest --cov --cov-report=html:coverage_report
to exclude a code block from coverage report, add
# pragma: no cover

=== Sleep / Wait for input ===
sleep for a while
import time
time.sleep(60)

wait for user input
input("press Enter to close")

===Suppress Warnings===
see [https://docs.python.org/3/library/warnings.html#temporarily-suppressing-warnings]
import warnings
warnings.filterwarnings("ignore", message=".*native_field_num.*not found in message.*")

===Logging===
====V4 minimal stdout logging====
import logging
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s")
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
logging.getLogger("httpx").setLevel(logging.WARNING)

# usage of variables and rounding of numbers. (%d also rounds floats)
LOGGER.info("file %s has %d lines with avg %.1f char/line", filename, lines, c_p_l)

====V3 simple logging====
# main file:
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(name)s\t%(message)s",
handlers=[
RotatingFileHandler(
Path(__file__).with_suffix(".log"),
maxBytes=10485760, # 10 MB = 10*1024*1024,
backupCount=1,
encoding="utf-8",
)
],
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(Path(__file__).stem)
logger.info("Start")

# other files:
import logging
logger = logging.getLogger(__name__)

====V2: rotating file and STDOUT====
# 1. setup
import logging
from logging.handlers import RotatingFileHandler

logfile = "myApp.log"
maxBytes = 20 * 1024 * 1024
backupCount = 5
loglevel_console = logging.INFO
loglevel_file = logging.DEBUG

# create logger
logger = logging.getLogger("root")
logger.setLevel(loglevel_file)

# console handler
ch = logging.StreamHandler()
ch.setLevel(loglevel_console)

# rotating file handler
# fh = logging.FileHandler(logfile)
fh = RotatingFileHandler(logfile, maxBytes=maxBytes, backupCount=backupCount)
fh.setLevel(loglevel_file)

# create formatter and add it to the handlers
# %(name)s = LoggerName, %(threadName)s = TreadName
formatter = logging.Formatter(
"%(asctime)s - %(levelname)s - %(name)s - %(threadName)s - %(message)s "
)
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(ch)
logger.addHandler(fh)

logger.debug("DebugMe")
logger.info("Starting")
logger.warning("Attention")
logger.error("Something went wrong")
logger.critical("Something seriously went wrong ")

# 2. in other files/modules now use
import logging

logger = logging.getLogger(__name__)
logger.info("text")

...
except Exception as e:
logger.exception("Unhandeled exception")
quit()

====V1: Simple====
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(message)s", # \t%(name)s
filename=Path(__file__).with_suffix(".log"), # uncomment to switch to stdout
filemode="a",
)
logger = logging.getLogger(__name__)
logger.debug("Some text")
logger.info("Some text")
logger.warning("Some text")
logger.error("Some text")
logger.critical("Some text")
try:
...
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise

===Process Bar===
see [https://tqdm.github.io tqdm]
from tqdm import tqdm
for i in tqdm(range(10000)):
....
or
with tqdm(total=total_iterations, desc="Processing") as progress_bar:
...
progress_bar.update(1)

===CGI Web development===
# Print necessary headers.
print("Content-Type: text/html")
print()

# errors and debugging info to browser
import cgitb
cgitb.enable()

Access URL or Form Parameters
# V2 from https://www.tutorialspoint.com/python/python_cgi_programming.htm
import cgi
form = cgi.FieldStorage()
username = form.getvalue('username')
print(username)

# V1
import sys
import urllib.parse
query = os.environ.get('QUERY_STRING')
query = urllib.parse.unquote(query, errors="surrogateescape")
d = dict(qc.split("=") for qc in query.split("&"))
print(d)

====CGI Backend Returning JSONs====
#!/usr/local/bin/python3.6
# -*- coding: utf-8 -*-

import cgi
import json

# Print necessary headers.
print("Content-type: application/json")
print()

def get_form_parameter(para: str) -> str:
"asserts that a given parameter is set and returns its value"
value = form.getvalue(para)
assert value, f"Error: parameter {para} missing"
assert value != "", f"Error: parameter {para} missing"
return value

response = {}
response['status'] = "ok"

try:
action = get_form_parameter("action")
response['action'] = action
if action == "myAction":
...

except Exception as e:
response['status'] = "error"
d = {"type": str(type(e)), "text": str(e)}
response["exception"] = d

finally:
print(json.dumps(response))

==Databases==
===PostgreSQL===
====Improved helper function====
"""Helper functions for DB access."""

import datetime as dt
import logging

import psycopg2
import psycopg2.extras
from creds_db import credentials as creds_db

# Configure logging
logger = logging.getLogger(__name__)

connection: psycopg2.extensions.connection = None # type: ignore
cursor_query: psycopg2.extensions.cursor = None # type: ignore
cursor_write: psycopg2.extensions.cursor = None # type: ignore

def connect() -> (
tuple[
psycopg2.extensions.connection,
psycopg2.extensions.cursor,
psycopg2.extensions.cursor,
]
):
"""Connect to the database."""
try:
connection = psycopg2.connect(**creds_db)
cursor_query = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)
cursor_write = connection.cursor()
logger.info(
"Connected to database %s on host %s",
creds_db["database"],
creds_db["host"],
)
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise
else:
return connection, cursor_query, cursor_write

def disconnect() -> None:
"""Disconnect from the database."""
try:
if cursor_query:
cursor_query.close()
if cursor_write:
cursor_write.close()
if connection:
connection.close()
logger.info("Disconnected from the database")
except psycopg2.DatabaseError:
logger.exception("Error during disconnection: %s")

def exists(url: str, valid_hours: float) -> bool | None:
"""
Check if a record exists.

Returns
-------
None for missing record
True for valid record
False for expired record

"""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=valid_hours)
sql = """
SELECT delete_at > %s AS "valid"
FROM my_table
WHERE url = %s LIMIT 1;
"""
cursor_query.execute(sql, (delete_at, url))
res = cursor_query.fetchone()
return res[0] if res else None

def query1(url: str) -> dict | None:
"""Execute a query."""
sql = """
SELECT *
FROM my_table
WHERE url = %s LIMIT 1
"""
cursor_query.execute(sql, (url,))
return cursor_query.fetchone() # type: ignore

def insert(url: str, response: str, delete_in_hours: float) -> None:
"""Insert a record."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=delete_in_hours)
sql = """
INSERT INTO my_table (url, response, delete_at)
VALUES (%s, %s, %s);
"""
cursor_write.execute(sql, (url, response, delete_at))
connection.commit()

def delete(url: str) -> None:
"""Delete a record."""
sql = """
DELETE FROM my_table
WHERE url = %s;
"""
cursor_write.execute(sql, (url,))
connection.commit()

def delete_expired(in_hours: float) -> None:
"""Delete records that expire in less than in_hours from now."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=in_hours)

sql = """
DELETE FROM my_table
WHERE delete_at < %s;
"""
cursor_write.execute(sql, (delete_at,))
connection.commit()

connection, cursor_query, cursor_write = connect()

====Basics====
import psycopg2
import psycopg2.extras

credentials = {
"host": "localhost",
"port": 5432,
"database": "myDB",
"user": "myUser",
"password": "myPwd",
}
connection = psycopg2.connect(**credentials)
cursor = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)

l_bind_vars = [["A1", "A2"], ["B1", "B2"]]

sql = """
SELECT * FROM myTable
WHERE 1=1
AND status NOT IN ('CLOSED')
AND ColA = %s
AND ColB = %s
ORDER BY created DESC
"""
cursor.execute(sql, l_bind_vars)
d_data = dict(cursor.fetchone())

====export result to csv file====
sql1 = "SELECT * FROM table"
sql2 = "COPY (" + sql1 + ") TO STDOUT WITH CSV HEADER DELIMITER '\t'"
with open("out.csv", "w") as file:
cursor.copy_expert(sql2, file)

====ReadConfigFile to connect to PostgreSQL====
from [http://www.postgresqltutorial.com/postgresql-python/connect/]
database.ini
[postgresql]
host=dbhost
port=5432
database=dbname
user=dbuser
password=dbpass

from configparser import ConfigParser

def config(filename="database.ini", section="postgresql"):
parser = ConfigParser()
parser.read(filename)
# get section, default to postgresql
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception(
"Section {0} not found in the {1} file".format(section, filename)
)
return db

main.py
import psycopg2
from config import config

def connect():
"""Connect to the PostgreSQL database server"""
conn = None
try:
params = config() # read connection parameters
print("Connecting to the PostgreSQL database...")
conn = psycopg2.connect(**params)
cur = conn.cursor() # create a cursor
print("PostgreSQL database version:")
cur.execute("SELECT version()") # execute a statement
db_version = cur.fetchone()
print(db_version)
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
print("Database connection closed.")

if __name__ == "__main__":
connect()

===SQL Lite / SQLite===
see page [[SQLite]]

===InfluxDB===
see [[InfluxDB]]

==Internet Access==
===Send E-Mails===
see [[Python - eMail]]

===Download data using browser UA / REST===
import requests

headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0",
}
resp = requests.get(
url,
headers=headers,
timeout=3, # timeout in sec, requests should always have a timeout!
# timeout=(1,3), # 1s connect-timout, 3s read-timeout
)
if resp.status_code != 200: # noqa: PLR2004
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download only if cache is too old====
import time
from pathlib import Path
import requests

def fetch_url_or_cache(file_cache: Path, url) -> str:
"""Fetch URL and cache in a file."""
if check_cache_file_available_and_recent(
file_path=file_cache, max_age=3600, verbose=False
):
with file_cache.open(mode="r", encoding="utf-8") as fh:
s = fh.read()
else:
s = fetch(url=url)
with file_cache.open(mode="w", encoding="utf-8", newline="\n") as fh:
fh.writelines(s)
return s

def check_cache_file_available_and_recent(
file_path: Path,
max_age: int = 3500,
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = False
if file_path.exists() and (time.time() - file_path.stat().st_mtime < max_age):
cache_good = True
return cache_good

# or
def check_cache_file_available_and_recent_verbose(
file_path: Path, max_age: int = 3600, *, verbose: bool = False
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = True
if not file_path.exists():
if verbose:
print(f"No Cache available: {file_path}")
cache_good = False
if cache_good and time.time() - (time.time() - file_path.stat().st_mtime < max_age):
if verbose:
print(f"Cache too old: {file_path}")
cache_good = False
return cache_good

def fetch(url) -> str:
"""Fetch URL."""
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0", # noqa: E501
}
resp = requests.get(url, headers=headers, timeout=5)
if resp.status_code == 200: # noqa: PLR2004
return resp.content.decode("ascii")
else: # noqa: RET505
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download using urllib.request====
from urllib.request import urlopen

url = "https://pomber.github.io/covid19/timeseries.json"
with urlopen(url) as response: # noqa: S310
response_text = response.read()

===HTML parsing and extracting of elements===
V2: via BeautifulSoup
from bs4 import BeautifulSoup # pip install beautifulsoup4

soup = BeautifulSoup(cont, features="html.parser")
my_element = soup.find("div", {"class": "user-formatted-inner"})
my_body = my_element.prettify()
# my_body = my_element.encode()
# my_body = str(my_element)

V1: via lxml and xpath
from lxml import html
import requests

page = requests.get(url)
tree = html.fromstring(page.content)
tbody_trs = tree.xpath("//*/tbody/tr")
l_rows = []
for tr in tbody_trs:
l_columns = []
if len(tr) != 15:
continue
for td in tr:
l_columns.append(td.text_content())
l_rows.append(list(l_columns))

===HTML entities to unicode===
import html
cont = html.unescape(cont)

==GUI Interactions==
===Take Screenshot===
import pyautogui # (c:\Python\Scripts\)pip install pyautogui
# pyautogui does only support screenshots on monitor #1
...
screenshot = pyautogui.screenshot()
# screenshot = pyautogui.screenshot(region=(screenshotX,screenshotY, screenshotW, screenshotH))
screenshot = np.array(screenshot)
# Convert RGB to BGR
screenshot = screenshot[:, :, ::-1].copy()

===Mouse Actions===
def clickIt(x,y,key="") :
x0, y0 = pyautogui.position()
if key != "": # crtl, shift
pyautogui.keyDown(key)
pyautogui.moveTo(x, y, duration=0.2)
pyautogui.click(x=x , y=y, button='left', clicks=1, interval=0.1)
if key != "": # crtl, shift
pyautogui.keyUp(key)
pyautogui.moveTo(x0, y0)

===Web Automation===
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time
import glob

class StravaUserMapDL():
def __init__(self):
self.driver = webdriver.Firefox()

def login(self):
driver = self.driver
url = "https://www.somewebpage.com"
email = "myemail"
password = "mypassword"
driver.get(url)

title = driver.title
urlIs = driver.current_url
cont = driver.page_source # as string
FILE = open(filename,"w") # w = overWrite file ; a = append to file
FILE.write(cont)
FILE.close()

# handle login if urlIs != url
if (urlIs != url):
# activate checkbox 'remember_me'
elem = driver.find_element_by_id('remember_me')
if (elem.is_selected() == False):
elem.click()
assert elem.is_selected() == True
elem = driver.find_element_by_id('email')
elem.send_keys(email)
elem = driver.find_element_by_id('password')
elem.send_keys(password)
elem.send_keys(Keys.RETURN)
# Wait until login pages is replaced by real page
urlIs = driver.current_url
while (urlIs != url):
time.sleep(1)
urlIs = driver.current_url
print (urlIs)

# results = driver.find_elements_by_class_name('following')
# results = driver.find_elements_by_tag_name('li')

# print(results[0].text)
assert (urlIs == url)

===Unit Tests using Web Automation===
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time

class PythonOrgSearch(unittest.TestCase):
# def __init__(self,asdf):
# self.driver = webdriver.Firefox()

def setUp(self):
print ("setUp")
# headless mode:
# opts = Options()
# opts.set_headless()
# assert opts.headless # Operating in headless mode
# self.driver = webdriver.Firefox(options=opts)

self.driver = webdriver.Firefox()

def test_search_in_python_org(self):
driver = self.driver
driver.get("http://www.python.org")
self.assertIn("Python", driver.title)
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
print ("fertig: python_org")

def tearDown(self):
print ("tearDown")
print ("close Firefox")
self.driver.close() # close tab
self.driver.quit() # quit browser
# os._exit(1) # exit unittest without Exception

if __name__ == "__main__":
try:
unittest.main()
except SystemExit as e:
os._exit(1)

==Cryptography and Hashing==
====Hashing via SHA256====
def gen_SHA256_string(s: str) -> str:
m = hashlib.sha256()
m.update(s.encode("ascii"))
return m.hexdigest()

====Hashing via MD5====
(MD5 is not secure, better use SHA256)
def gen_MD5_string(s: str) -> str:
m = hashlib.md5()
m.update(s.encode("ascii"))
return m.hexdigest()

====Password hashing via bcrypt====
import bcrypt
pwd = 'geheim'
pwd = pwd.encode("utf-8")
# or
pwd = b'geheim'

hashed = bcrypt.hashpw(pwd, bcrypt.gensalt())
if bcrypt.checkpw(pwd, hashed):
print("It Matches!")
print(hashed.decode("utf-8"))

To use version 2a instead of 2b (default):
bcrypt.gensalt(prefix=b"2a")

==Multiprocessing, subprocesses and Threading==
see [[Python_-_Multithreading]] as well

use processes for CPU limited work 
use threads for I/O limited work

===Simple single process===
import subprocess
process = subprocess.run(["sudo", "du", "--max-depth=1", mydir], capture_output=True, text=True)
print (process.stdout)
old, depricated way:
os.system( "gnuplot " + gpfile)

===Multiprocessing===
see [[Python - Multithreading]] as well

V2 using pool and starmap
import multiprocessing
import os

def worker(i: int, s: str) -> list:
result = (i, s, os.getpid())
return result

if __name__ == "__main__":
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n" + str(i))
l_pile_of_work.append((tup))
# gen pool of processes
num_processes = min(multiprocessing.cpu_count(), len(l_pile_of_work))
pool = multiprocessing.Pool(processes=num_processes)
# start processes on pile of work
l_results_unsorted = pool.starmap(
func=worker, iterable=l_pile_of_work # each item is a list of 2 parameters
)

# or if only one parameter:
# l_results_unsorted = pool.map(doit_de_district, l_pile_of_work)

l_results = sorted(l_results_unsorted) # sort by i

V1
import subprocess
l_subprocesses = [] # queue list of subprocesses
max_processes = 4

def process_enqueue(new_process_parameters):
global l_subprocesses
# wait for free slot
while len(l_subprocesses) >= max_processes:
process_remove_finished_from_queue()
time.sleep(0.1) # sleep 0.1s
process = subprocess.Popen(new_process_parameters,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
l_subprocesses.append(process)

def process_remove_finished_from_queue():
global l_subprocesses
i = 0
while i <= len(l_subprocesses) - 1:
process = l_subprocesses[i]
if process.poll != None: # has already finished
process_print_output(process)
l_subprocesses.pop(i)
else: # still running
i += 1

def process_print_output(process):
"""waits for process to finish and prints process output"""
stdout, stderr = process.communicate()
if stdout != '':
print(f'Out: {stdout}')
if stderr != '':
print(f'ERROR: {stderr}')

def process_wait_for_all_finished():
global l_subprocesses
for process in l_subprocesses:
process_print_output(process)
l_subprocesses = [] # empty list of done subprocesses

process_enqueue(l_parameters1)
...
process_enqueue(l_parameters999)
process_wait_for_all_finished()

===Threading===
import threading
import queue
import os
import time

def worker(q_work: queue.Queue, results: dict):
while not q_work.empty():
i, s = q_work.get()
time.sleep(.1)
result = (i, s, os.getpid())
results[i] = result
q_work.task_done()

if __name__ == '__main__':
d_results = {} # threads can write into dict
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n"+str(i))
l_pile_of_work.append((tup))
# convert list of work to queue
q_pile_of_work = queue.Queue(
maxsize=len(l_pile_of_work)) # maxsize=0 -> unlimited
for params in l_pile_of_work:
q_pile_of_work.put(params)
# gen threads
num_threads = 100
l_threads = [] # List of threads, not used here
for i in range(num_threads):
t = threading.Thread(name='myThread-'+str(i),
target=worker,
args=(q_pile_of_work, d_results),
daemon=True)
l_threads.append(t)
t.start()
q_pile_of_work.join() # wait for all threas to complete
l_results_unsorted = d_results.values()
l_results = sorted(l_results_unsorted) # sort by i

===asyncio — Asynchronous I/O===
see https://docs.python.org/3/library/asyncio-task.html
import asyncio
import time

# basics
# task = asyncio.create_task(coro())
# Wrap the coro coroutine into a Task and schedule its execution. Return the Task object.

# sleep
# await asyncio.sleep(1)

# Running Tasks Concurrently and gathers the return values in list L
# L = await asyncio.gather(coro(x1,y1), coro(x2,y2), coro(x3,y3))

async def say_after(delay, what):
# Coroutines declared with the async/await syntax
await asyncio.sleep(delay)
print(what)

async def main():
# The asyncio.create_task() function to run coroutines concurrently as asyncio Tasks.
task1 = asyncio.create_task(
say_after(1, 'hello'))

task2 = asyncio.create_task(
say_after(1, 'world'))

print(f"started at {time.strftime('%X')}")

# Wait until both tasks are completed
await task1
await task2

print(f"finished at {time.strftime('%X')}")

asyncio.run(main())

==Pandas==
see [[Pandas]]

==Matplotlib==
see [[Matplotlib]]

== GUI via tkinter==
import tkinter as tk # no need to install via pip

class App(tk.Tk):
def __init__(self):
super().__init__()
self.title("robot-CClicker")
self.geometry("200x200+0+1080")
self.resizable(width=False, height=False)

self.l_buttons = []
self.__create_widgets()

def __create_widgets(self):
self.btn_click500 = tk.Button(
master=self,
text="500 clicks",
width=20,
command=lambda: self.clickBigCookie(500),
)
self.l_buttons.append(self.btn_click500)

for button in self.l_buttons:
button.pack(anchor=tk.W)

def clickBigCookie(self, num):
# TODO: the disabling of the buttons is not working
for button in self.l_buttons:
button["state"] = tk.DISABLED
helper.clickIt(self.posBigCockie[0], self.posBigCockie[1], num=num)
for button in self.l_buttons:
button["state"] = tk.NORMAL

if __name__ == "__main__":
app = App()
app.mainloop()

==Protobuf==
===convert .proto file to python class===
see https://www.datascienceblog.net/post/programming/essential-protobuf-guide-python/
protoc my_message.proto --python_out ./

===read/decode protobuf message===
parse message from file
import machine_message_pb2

with open("out.bin", "rb") as f:
my_message = my_message_pb2.my_message()
my_message.ParseFromString(f.read())
print(machine_message)

===create/encode protobuf message===
import machine_message_pb2

my_message = my_message_pb2.my_message()
my_message.data.field1 = 1
my_message.data.field2 = "asdf"

with open("out.bin", "wb") as f:
f.write(my_message.data.SerializeToString())

==venv / virtual environments==
pip install --upgrade pip
python -m venv .venv --prompt $(basename $(pwd))
source .venv/bin/activate
pip install -r requirements.txt
...
deactivate

==Pydantic data validation==
===Function input parameter validation===
see https://docs.pydantic.dev/usage/validation_decorator/
@validate_arguments
def sum(x: int, y: float) -> float:
print(x, y)
return x + y

print(sum(1.1, 1.1))
# -> 2.1

==Read config file==
===TOML===
see https://learnxinyminutes.com/docs/toml/

minimal example:
config.toml
# SAP API endpoint
[general]
sleep_time = 60
use_proxy = true
tool.py
try:
import tomllib # comes with python3.11
except ModuleNotFoundError:
import tomli as tomllib # pip install tomli
with open("config.toml", "rb") as f:
o = tomllib.load(f) # type: ignore
if o["settings"]["use_proxy"]:
...
for validation, see https://realpython.com/python-toml/

====TOML Write====
In for deployments I sometime want to modify a TOML config file for production, based on the dev version.
import tomli_w # pip install tomli-w
p_in = Path(__file__).parent.parent / ".streamlit/config.toml"
p_out = p_in.parent / "config-prod.toml"
with p_in.open("rb") as fh:
o = tomllib.load(fh)
o["server"]["fileWatcherType"] = "none"
del o["server"]["address"]
with p_out.open("wb") as fh:
tomli_w.dump(o, fh)

===ConfigParser: INI Config File Reading===
better use TOML

Config.ini
[Section1]
Cursor = 205E18
Grandma = 18E18
Farm = 11E18
Mine = 514E18
Factory = 155E18
test.py
from configparser import ConfigParser

config = ConfigParser(
interpolation=None
) # interpolation=None -> treats % in values as char % instead of interpreting it
config.read("Config.ini", encoding="utf-8")

print(config.getint("Section1", "key1"))
print(config.getfloat("Section1", "key2"))
print(config.get("Section1", "key3"))

for sec in config.sections():
d_settings = {}
for key in config.options(sec):
value = config.get(sec, key)
d_settings[key] = value
print("%15s : %s" % (key, value))

==VCard / vcf==
import codecs
import vobject # pip install vobject

obj = vobject.readComponents(codecs.open(file_in, encoding="utf-8").read()) # type: ignore
contacts: list[vobject.base.Component] = list(obj) # type: ignore

card = contacts[0]

if "bday" not in card.contents:
continue

# bday: remove 'VALUE': ['DATE']
card.contents["bday"][0].params = {} # type: ignore

# remove all fields but "bday", "n"
for key in card.contents.copy(): # loop over copy, to allow for deleting keys
if key not in ("n", "bday"):
del card.contents[key]

# recreate fn based on n
card.add("fn")
n = card.contents["n"][0]
fn = f"{n.value.given} {n.value.additional} {n.value.family}" # type: ignore
fn = re.sub(r"\s+", " ", fn)
card.fn.value = fn # type: ignore

with open("out.vcf", mode="w", encoding="utf-8", newline="\n") as fhOut:
fhOut.write(card.serialize())

==Sentry Exception monitoring==
see [[Sentry]]
==MQTT==
#!/usr/bin/env python3.12
"""Read power data from Tasmota device via MQTT."""

import json
from typing import Any

import paho.mqtt.client as mqtt
from mqtt_credentials import hostname, password, port, username
from paho.mqtt.reasoncodes import ReasonCode

def on_connect(
mqtt_client: mqtt.Client,
userdata: Any,
flags: dict[str, int],
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client connects MQTT broker.""" # noqa: D401
if reason_code == 0:
print("Connected to MQTT")
# print(f"MQTT protocol version: {mqtt_client._protocol}")

# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
mqtt_client.message_callback_add("tele/tasmota_MT681/SENSOR", on_message)
mqtt_client.subscribe([("tele/tasmota_MT681/SENSOR", 0)])

else:
print(f"Connection to MQTT broker failed. Error: {reason_code}")

def on_disconnect(
mqtt_client: mqtt.Client,
userdata: Any,
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client is disconnected from the MQTT broker.""" # noqa: D401
if reason_code > 0:
print(f"Unexpected disconnection. Error: {reason_code}")

def on_message(
mqtt_client: mqtt.Client,
userdata: Any,
message: mqtt.MQTTMessage,
) -> None:
"""Callback when a Tasmota message is received from the MQTT broker.""" # noqa: D401
s = message.payload.decode()
data = json.loads(s)
# {'Time': '2024-03-16T06:44:08', 'MT681': {'Total_in': 16.4396, 'Power_cur': 80, 'Total_out': 14.6403}} # noqa: E501

total_i = data["MT681"]["Total_in"]
total_o = data["MT681"]["Total_out"]

print(f"In:\t{total_i}\nOut:\t{total_o}")

mqtt_client.disconnect()
mqtt_client.loop_stop()

try:
# Create an MQTT client instance
mqtt_client: mqtt.Client = mqtt.Client(
mqtt.CallbackAPIVersion.VERSION2, "Python-Client-1"
) # type: ignore
mqtt_client.username_pw_set(username, password)

mqtt_client.on_connect = on_connect

# Connect to the MQTT broker
mqtt_client.connect(hostname, port, keepalive=60)

# NOT: while True, as that eats the CPU !!!
mqtt_client.loop_forever()
except KeyboardInterrupt:
print("KeyboardInterrupt, disconnecting from MQTT broker.")
mqtt_client.disconnect()
mqtt_client.loop_stop()

==Caching Functions==
from functools import lru_cache

@lru_cache(maxsize=128) # None equals to @cache declarator
def fnc(n:int):

==Performance/Resources==
===RAM/Memory consumption/bottleneck===
import tracemalloc
tracemalloc.start()

# ...
# Code
# example:
lst = list(range(1_000_000))
# ...

max_bytes = tracemalloc.get_traced_memory()[0]
print(f"{round(max_bytes / 10**6)}MB")

tracemalloc.stop()

===Profiling / Count and Runtime per Function===
call_stats = {}

def track_function_usage(func: Callable) -> Callable:
"""Annotation for gathering runtime statistics."""

@wraps(func)
def wrapper(*args: tuple, **kwargs: dict): # noqa: ANN202
start_time = time.time()
result = func(*args, **kwargs) # Call the original function
end_time = time.time()
call_stats[func.__name__]["calls"] += 1
call_stats[func.__name__]["total_time"] += end_time - start_time
return result
return wrapper

==JWT Token==
try:
decoded_token = jwt.decode(token, options={"verify_signature": False})
except jwt.ExpiredSignatureError:
msg = "Token has expired."
logger.exception(msg)
except jwt.InvalidTokenError:
msg = "Invalid token."
logger.exception(msg)

==Streamlit==
see https://github.com/entorb/streamlit-examples for a collection of my examples

===Minimum Example===
# src/app.py
from pathlib import Path
import streamlit as st
from streamlit.logger import get_logger
logger = get_logger(Path(__file__).stem)
logger.info("Start")
st.set_page_config(page_title="AppTitle", page_icon=None, layout="wide")
st.title("MyPageTitle")
st.header("MyHeader")
st.subheader("MySubHeader")
sel_param = st.selectbox("Parameter", ("count", "sum"))

# .streamlit/config.toml
[browser]
gatherUsageStats = false

[server]
port = 8501

# run it
streamlit run src/app.py

====Reduce Logging for K8s Deployed Docker Containers====
# reduce log output
for logger_name in [
"streamlit.runtime.caching.cache_utils",
"streamlit.runtime.caching.storage.in_memory_cache_storage_wrapper",
"streamlit.runtime.runtime",
"urllib3.connectionpool",
]:
get_logger(logger_name).setLevel(logging.WARNING)

===Chart via Altair===
====Line Chart====
chart = st.line_chart(df["value"])
# corresponds to
import altair as alt
c = (
alt.Chart(df)
.mark_line()
.encode(
x=alt.X("created", title=None),
y=alt.Y("value", title=None),
)
)
st.altair_chart(c, use_container_width=True)

====Bar Chart====
chart = (
alt.Chart(df)
.mark_bar()
.encode(
x=alt.X("yearmonth(date):T", title="Month"),
y=alt.Y(f"count:Q", title=None),
color="status:N",
tooltip=["yearmonth(date):T", "status:N", "cnt:Q"],
)
# .properties(width=600, height=400)
)
st.altair_chart(chart, use_container_width=True)

:T stands for Temporal. It indicates that the field contains date or time values.
:N stands for Nominal. It indicates that the field contains categorical data, which represents discrete categories or labels.
:Q stands for Quantitative. It indicates that the field contains numerical data that can be measured and compared.

===Excel Download===
def excel_download_buttons(df: pd.DataFrame, file_name: str = "export.xlsx") -> None:
"""Show prepare data and download buttons."""
if st.button(label="Click to prepare download"):
buffer = io.BytesIO()
with pd.ExcelWriter(buffer, engine="xlsxwriter") as writer:
df.to_excel(writer, sheet_name="Sheet1", index=False)
writer.close()

st.download_button(
label="Download data as Excel",
data=buffer,
file_name=file_name,
mime="application/vnd.ms-excel",
)

==UV Package Manager==
# create/update lockfile
uv lock --upgrade
# sync venv
uv sync --upgrade
# extract package info to requirements.txt
uv export --format requirements.txt --no-dev --no-hashes -o requirements.txt

Update Python version for my current project
uv venv --python=3.13.7

===Update UV===
uv self update
# or
brew update && brew upgrade uv

===Update Python Versions===
uv python upgrade

===UV + Docker on readOnlyRootFilesystem===
Example 1: https://github.com/SWE-Group-23/CM22007---Source/blob/10d02004f3a4208f7b30d37a0a87e89532a23575/create-service.sh#L117

Dockerfile
WORKDIR /app
COPY . .

ENV CASS_DRIVER_NO_EXTENSIONS=1

RUN --mount=type=cache,target=/root/.cache/uv \\
--mount=type=bind,source=uv.lock,target=uv.lock \\
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \\
uv sync --frozen --no-install-project

RUN uv pip install -e shared/

RUN addgroup -S group1 && adduser -S user1 -G group1 -u 1000
RUN chown -R user1:group1 /app
USER user1:group1

CMD ["uv", "run", "main.py"]

Deployment
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /home/user1/.cache/uv
name: uv
- mountPath: /tmp
name: temp

Example 2: [https://hynek.me/articles/docker-uv/ Production-ready Python Docker Containers with uv]

Dockerfile (without the comments)
ENV UV_LINK_MODE=copy \
UV_COMPILE_BYTECODE=1 \
UV_PYTHON_DOWNLOADS=never \
UV_PYTHON=python3.12 \
UV_PROJECT_ENVIRONMENT=/app

RUN --mount=type=cache,target=/root/.cache \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync \
--locked \
--no-dev \
--no-install-project

COPY . /src
WORKDIR /src
RUN --mount=type=cache,target=/root/.cache \
uv sync \
--locked \
--no-dev \
--no-editable

== Vulture: Search for dead code ==
see [https://github.com/jendrikseipp/vulture GitHub]

pyproject.toml
[tool.vulture]
paths = ["src"]
# exclude = ["src/models.py"]
ignore_names = [
"LOGGER",
]

.pre-commit-config.yaml
# Vulture: find dead code
- repo: https://github.com/jendrikseipp/vulture
rev: v2.16
hooks:
- id: vulture

run
pip install vulture
vulture src/

uv add --dev vulture
uv run vulture src/

Python

2026-04-12T12:31:06Z

Torben: /* Vulture: Search for dead code */

[[Category:Coding]][[Category:Python]]

==Getting Started==
===Install===
====Python====
Best use [https://docs.astral.sh/uv/ UV] for managing python and package versions.

* for Windows: get and install Python from https://www.python.org
* for MacOS: use [https://github.com/pyenv/pyenv?tab=readme-ov-file#a-getting-pyenv pyenv]
brew install xz
brew install pyenv

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[ [ -d $PYENV_ROOT/bin ] ] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init - zsh)"' >> ~/.zshrc
exec "$SHELL"

pyenv install --list
pyenv install 3.13.5
pyenv global 3.13.5

vim ~/.zshrc
# add
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi

====Editor: Visual Studio Code====
excellent and free source-code editor that supports many languages.
* get and install from https://code.visualstudio.com

See Wickie page [[Visual_Studio_Code]] for general setup, extension, config...

Python Extensions
* Python
* Pylance
* [https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff Ruff Formater and Linter]
* [https://marketplace.visualstudio.com/items/?itemName=tamasfe.even-better-toml Even Better TOML]
Settings (CTRL + ,)
* Extensions -> Python -> Formatting: Provider = black
* Text Editor -> Editor: Format On Save
* Text Editor -> Files:Eol -> \n
* Linter: Ruff
Settings in settings.json
"python.analysis.completeFunctionParens": true,
"python.analysis.autoImportCompletions": true,
"python.analysis.inlayHints.functionReturnTypes": true,
"python.analysis.typeCheckingMode": "strict",
// Ruff
"[python]": {
"editor.defaultFormatter": "charliermarsh.ruff",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.fixAll": "explicit",
"source.organizeImports.ruff": "explicit"
},
},

Run your code
CTRL + F5 : run
F5 : run in debugger

====Code Formatter Ruff====
use software for handling the code formatting like "ruff" or "black", where Ruff is much faster and additionally provides linting.
pip install ruff
than activate in editor like vs code

===My Template===
see latest version at https://github.com/entorb/template-python

"""Main file."""
# by Torben Menke https://entorb.net
import logging
from pathlib import Path

def init_logging() -> None:
"""Initialize logging."""
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(
level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s"
)
logging.getLogger("httpx").setLevel(logging.WARNING)

init_logging()
LOGGER = logging.getLogger(__name__)

def main():
LOGGER.info("Moin Moin")

if __name__ == "__main__":
main()

"""Other file"""
import logging
LOGGER = logging.getLogger(__name__)

===Compile to .exe===
pip install pyinstaller
pyinstaller --onefile --console your.py
# for Excel and Matplotlib these options are required
--hidden-import=openpyxl --hidden-import=matplotlib --hidden-import pandas.plotting._matplotlib
([[Python - py2exe]] is deprecated)

==Basics==
=== Naming Conventions===
[https://google.github.io/styleguide/pyguide.html Google Python Style Guide]:
module_name, package_name, ClassName, method_name, ExceptionName, function_name, GLOBAL_CONSTANT_NAME, global_var_name, instance_var_name, function_parameter_name, local_var_name

====Doc Strings====
It is best practice to start a file with a docstring:
"""My Script"""
This can be accessed linke this:
title = __doc__[:-1] # type: ignore

===Installing packages===
python -m pip install --upgrade pip

pip install somemodule
# or
pip3 install somemodule
# or read from file
pip install -r requirements.txt

# uninstall
pip uninstall somemodule

# using a web proxy
# set proxy for windows cmd session
SET HTTPS_PROXY=http://myProxy:8080
(afterwards --proxy setting below no longer required
or
pip install --proxy http://myProxy:8080 somemodule

# list outdated packages
pip list --outdated

# update package
pip install --upgrade pyinstaller

# updating all via Windows Powershell (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | %{$_.split('==')[0]} | %{pip install --upgrade $_}

# updating all via Bash (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install --upgrade

# downgrade
pip install --upgrade pandas==1.2.4

====Oneline Updater for requirements.txt====
https://pkgui.com/pip

====update packages====
pip install pip-review
pip-review --auto

====generate requirements.txt====
pip install pipreqs
pipreqs ./

===Loops===
ATTENTION: The loops do not create a new variable scope. Only functions and modules introduce a new scope!
for p in all_files:
print(p)

for i, p in enumerate(all_files):
print(i, p)

for i in range(10):
print(i)
# del i

while i <= 100:
i += 1
...
if sth1:
continue # start next loop
if sth2:
break # exit loop

# inline if (requires a dummy else):
print("something") if sth else 0

===Math===
see [[Python - Math]] for linear regression

Modulo
15 % 4
# > 3

Integer Division
17 // 4
# > 4

====Random====
import random
random.randint(1000000, 9999999)

==Basic Objects==
===Variables===
del var # delete / undef a variable
var = None # sets to null

# check if variable is defined
if "var" in locals():
pass
# for object oriented projects:
if "var" in self.__dict__:
pass

# Access global variables in functions
var = 123

def test() -> None:
global var # point to global instead of creation of local var
var = 321

===Strings===
# num <-> str
s = str(i) # int to string
f = float(s) # str -> float
i = int(s)
str(round(f, 1)) # round first
# tests
s.isdigit() # 0-9
# note isdecimal() does also not match '1.1'

# printf: 1 digit
s = f"{value:0.1f}"
s = "%0.1f" % value

====Modify Strings====
# get string from prompt
s = input("Enter Text: ")

s = s.strip() # trim spaces from both sides, rstrip for right only
s = s.lower() # lower case
s = s.upper() # upper case
s = s.title() # upper case for first char of word

# upper case first letter of each word and also removes multiple and trailing spaces
import string
s = string.capwords(s)

# replace
s.replace(x, y)

# trim whitespaces from left and right
s.strip()

# replace all (multiple) whitespaces by single space ' '
s = " ".join(s.split())

# generate key value pairs from dict
# key1=value1&key2=value2
param_str = "&".join("=".join(tup) for tup in dict.items())

# repeat string multiple times
s * 5 # = s+s+s+s+s

====Substrings====
# find a substring:
if x in s: # True / False
pass
if len(s) > 0:
pass

# handling substrings
a = "abcd"
b = a[:1] + "o" + a[2:]
# > 'aocd'

s = "Hello there !bob@"
i1 = s.find("!") + 1
i2 = s.find("@")
substr = s[i1:i2]

def substr_between(s: str, s1: str, s2: str) -> str:
assert s1 in s, f"E: can't find '{s1}' in '{s}'"
assert s2 in s, f"E: can't find '{s1}' in '{s}'"
i1 = s.find(s1) + len(s1)
i2 = s.find(s2)
assert i1 < i2, f"E: '{s1}' not before '{s2}' in '{s}'"
return s[i1:i2]

====Binary, raw strings, html encoding====
# Binary Strings
sb = b"asdf"
# or
sb = str.encode("asdf")
s = sb.decode("ascii") # decode binary strings
sb = s.encode("ascii") # encode string to binary

# raw string
s = r"c:\Windows" # no escape of \ needed

# convert utf-8 to html umlaute
s = "Nürnberg".encode("ascii", "xmlcharrefreplace").decode()
# -> Nürnberg

====Guess encoding via chardet====
import chardet # pip install chardet
result = chardet.detect(raw_data)
if result["encoding"] and result["confidence"] > 0.5: # noqa: PLR2004
encoding = result["encoding"]
else:
encoding = "utf-8"

there is a much faster alternative [https://github.com/PyYoshi/cChardet cchardet], but that requires Microsoft Visual C++ 14.0

====Merge variables in string / sprintf====
print("Renner =", i)
print("Renner = %3d" % i) # leading 0's
print(f"Renner = {i}")

# place formatted numbers in a string / sprintf
s = "The {:.1f}% {} cost {:05.2f} euros".format( 5.1, "beer", 3.50)
print(s)
# > The 5.1% beer cost 03.50 euros

s = f"The length is {72.8958:.2f} meters"
# > The length is 72.90 meters

===Lists===
like @array in Perl
lst = [1, 2, 3, 4, 5, 6]
lst = [x / 2 for x in range(10)]
lst = [None] * 10 # Initiate list of None elements:
len(lst) # length
lst2 = lst[0:10] # get elements 0-10
for i in lst:
print(i)

====clone list====
# dangerous: creates a link to the original list
list_2 = my_list # M's elements are LINKS to L's
# clones can be achieved via:
list_2 = my_list.copy
list_2 = my_list[:]
list_2 = list(my_list)

====list modifications====
lst.pop() # returns and removes the last item
lst.pop(i) # returns and removes the item at position int i
lst.insert(i, x) # insert item x at position int i
lst.remove(x) # removes the first occurrence of item x
lst.append(x) # append a single element
lst.extend(m) # append elements of another list

lst.reverse() # reverse the order of the list
lst = sorted(["B", "a", "c"], key=str.casefold) # case insensitive / ignore case

# list to string
s = "".join(lst)
s = "\n".join(lst)
# string to list
lst = s.split() # default: split on space
lst = s.split(",")

# remove empty values from end
while L[-1] == "":
L.pop()

# check and remove items from list
# from https://stackoverflow.com/a/6024599
# iterates in-situ via reversed index
for i in range(len(lst) - 1, -1, -1):
element = lst[i]
if check(element):
del lst[i]

====check if item is in list====
x in lst
x not in lst
lst.count(x) # how many items x are in the list
i = lst.index("word") # find in list / returns the position of the first match in list

====pair 2 lists====
# zip: merge 2 lists to list of tuples
data = list(zip(data_x, data_y, strict=True))

# unzip: split list of pairs into 2 lists
data_x, data_y = zip(*data, strict=True)

# Cartesian product of lists / tuples
import itertools
for i in itertools.product(*list_of_lists):
print(i)

====sort multidim list====
lst = sorted(lst, key=lambda x: x[0], reverse=False)
lst = sorted(lst, key=lambda row: (row["Wann"], row ["Wer"]), reverse=False)

====filter list====
lines = [x for x in lines if not x.startswith ("word")]
lst = ["asdf", "asdf2", "qwertz"]
lst = [elem for elem in lst if "asdf" in elem]

====for each element====
# trim/strip spaces for each element
lines = [s.strip() for s in lines]
# modify each item in list by adding constant string
l = [s + ';' + v for v in l]

# modify item in list
for idx, line in enumerate(cont):
if "K1001/1" in line:
line = "K1001/1 Test Nr " + str(i) + "\n"
cont[idx] = line
break

===Tuples===
Ordered sequence, with no ability to replace or delete items
tpl = (1,2,3,4,5,6)

list -> tuple
tpl = tuple(lst)

combine 2 tuples
tpl = tpl1 + tpl2

===Dictionaries===
like %hash in Perl
d = {"key1": "value1", "key2": 123}
d["key3"] = (1, 2, 3)
del d["key3"]

len(d)
d.clear()
d.copy()
d.keys()
d.values()
d.items() # returns a list of tuples (key, value)
d.get(k) # returns value of key k
d.get(k, x) # returns value of key k; if k is not in d it returns x
count = d.get("key", 0) + 1 # nice for counters
d.pop(k) # returns and removes item k
d.pop(k, x) # returns and removes item k; if k is not in d it returns x
# checks
x in d
x not in d

# dict can have tuple as key
tpl = ("key1", "key2", 123)
d[tpl] = 123456

# loop over all key-value pairs
for key, value in d.items():
print(f"{key} = {value}")

# sorted keys:
for key in sorted(dict.keys()):
pass
# sorted values, reversed
for key, value in sorted(d.items(), key=lambda item: item[1], reverse=True):
pass

# join / merge 2 dicts
d.update(d2)

# flatten dict of dict
flat = d.copy()
meta = d.pop("metadata")
flat.update(meta)

====MultiDim Dictionaries====
d = {}
d["item1"] = {}
d["item1"]["value"] = 1.909e18
d["item1"]["name"] = "Item 1"
d["item2"] = {}
d["item2"]["value"] = 1.725e18
d["item2"]["name"] = "Item 2"

for k in d.keys():
d[k]["value"] = d[k]["value"] * 1.1

def find_common_strings(dict1: dict, dict2: dict) -> dict[Any, Any]:
"""Find common strings in two dict, returning a new dict of those strings."""

def recursive_match(d1: dict, d2: dict) -> dict[Any, Any]:
common_dict = {}
for key in d1: # noqa: PLC0206
if key in d2:
if isinstance(d1[key], dict) and isinstance(d2[key], dict):
# Recurse if both values are dicts
nested_common = recursive_match(d1[key], d2[key])
if nested_common: # Add to result if common values are found
common_dict[key] = nested_common
elif isinstance(d1[key], str) and isinstance(d2[key], str):
# Check if both are strings and identical
if d1[key] == d2[key]:
common_dict[key] = d1[key]
return common_dict

return recursive_match(dict1, dict2)

def update_dict_with_new_values(original_dict: dict, new_values: dict) -> dict:
"""Update strings in the original dict with new values from the new_values dict."""

def recursive_update(d1: dict, new_vals: dict) -> None:
for key in new_vals: # noqa: PLC0206
if key in d1:
if isinstance(d1[key], dict) and isinstance(new_vals[key], dict):
# Recurse if both values are dicts
recursive_update(d1[key], new_vals[key])
elif isinstance(d1[key], str) and isinstance(new_vals[key], str):
# Update the string
d1[key] = new_vals[key]

recursive_update(original_dict, new_values)
return original_dict

===Datetime, Date and Time===
import datetime as dt
from zoneinfo import ZoneInfo
TZ_DE = ZoneInfo("Europe/Berlin")
TZ_UTC = ZoneInfo("UTC")

====Create====
# date
date = dt.date(2023, 12, 31)
date = dt.date.fromisocalendar(int(year), int(week), int(daynum)) # daynum: 1..7
date = dt.date.fromisoformat("2020-03-10")
date = dt.datetime.strptime(datestr, "%y%m%d").date()
date_today = dt.datetime.now(tz=dt.UTC).date()
date_yesterday = dt.datetime.now(tz=TZ_DE).date() - dt.timedelta(days=1)
days_overdue = (date_completed - date_due).days
# to add one month (which is not supported by timedelta)
from dateutil.relativedelta import relativedelta
date_end = date_start + relativedelta(months=1)
delta_days = 1 + (end - start).days

# datetime
dt_now = dt.datetime.now(tz=TZ_DE)
my_dt = dt.datetime(2023, 12, 31, 14, 31, 56, tzinfo=TZ_DE) # 2023-12-31 14:31:56
my_dt = dt.datetime.fromtimestamp(my_timestamp, tz=TZ_DE)
my_dt = dt.datetime.fromisoformat ("2017-01-01T12:30:59.000000")
my_dt = dt.datetime.fromisoformat("2020-03-10 06:01:01+00:00")
s = "2020-03-10T06:01:01Z"
my_dt = dt.datetime.fromisoformat(s.replace("Z", " +00:00"))
my_date_local = my_dt.astimezone(tz=TZ_DE).date()

# datetime -> date
my_date = my_dt.date()
# date -> datetime
my_date = dt.date(2023, 12, 31)
my_dt = dt.datetime(my_date.year, my_date.month, my_date.day, tzinfo=TZ_DE)

# to string
date_str = my_dt.strftime("%y%m%d")
date_str = my_dt.strftime("%Y-%m-%d %H:%M:%S")
# alternative using time
date_str = time.strftime("%Y-%m-%d %H:%M:%S")
# now in UTC without milliseconds
date_str = dt.datetime.now(tz=dt.timezone.utc).replace(microsecond=0).isoformat() + "Z"

# remove timezone
my_dt = my_dt.replace(tzinfo=None)

# German format
import locale
locale.setlocale(locale.LC_ALL, "de_DE")
print(my_date.strftime("%a %x")) # Mo 25.12.2023

# Calendar week
week = my_date.isocalendar()[1]
print("KW%02d" % week)

# first day of quarter
def get_first_day_of_the_quarter(my_date: dt.date) -> dt.date:
return dt.date(my_date.year, 1 + 3 * ((my_date.month - 1) // 3), 1)

====rounding datetimes to minutes====
def floor_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Floor (=round down) minutes to X min resolution."""
min_new = res * (my_dt.minute // res)
return my_dt.replace(minute=min_new, second=0, microsecond=0)

def ceil_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Ceil (=round up) minutes to X min resolution."""
min_new = res * (1 + my_dt.minute // res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

def round_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Cound minutes to X min resolution."""
min_old_dec = float(my_dt.minute) + float(my_dt.second * 60)
min_new = res * round(min_old_dec / res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

my_dt = dt.datetime.fromisoformat("2011-11-04 00:05:23+00:00")
print(f"original: {my_dt}")
print(f"floored: {floor_dt_minutes(my_dt,5)}")
print(f"ceileded: {ceil_dt_minutes(my_dt,5)}")
print(f"rounded: {round_dt_minutes(my_dt,5)}")

====Timing / Time elapsed====
import time
timestart = time.time()
sec = time.time() - timestart
print(f"Time elapsed: {sec:.2f} sec")

==OS, Argpars, etc.==
===Hostname===
from socket import gethostname
print(gethostname())

===Checking Operating System===
import os
import sys

if os.name == "posix":
print("posix/Unix/Linux")
elif os.name == "nt":
print("windows")
else:
print("unknown os")
sys.exit(1) # throws exception, use quit() to close / die silently

===Get filename of python script===
my_file_path = __file__

alternative using sys package
from sys import argv
myFilename = argv[0]

===accessing os envrionment variables===
import os
print(os.getenv("tmp"))
# better:
try:
s = os.environ[key] # throws error if unset
except KeyError:
error_message(f"Environment variable {key} not set.")

===Command Line Arguments===
====ArgumentParser====
import argparse

parser = (
argparse.ArgumentParser()
) # construct the argument parser and parse the arguments
# -h comes automatically

# Boolean Parameter
parser.add_argument(
"-v", "--verbose", help="increase output verbosity", action="store_true"
) # store_true -> Boolean Value

# Choice Parameter
# restrict to a list of possible values / choices
# parser.add_argument("--choice", type=int, choices= [0, 1, 2], help="Test choices")

# Positional Parameter (like text.py 123)
# parser.add_argument("num", type=int, help="Number of things")

# Required Parameter
# parser.add_argument("-i", "--input", type=str, required=True, help="Path of file")

# Optional Parameter
parser.add_argument("-n", "--number", type=int, help="Number of clicks")
# Optional Parameter with Default
parser.add_argument(
"-s",
"--seconds",
type=int,
default=sec_default,
help="Duration of clicking, default = %i (sec)" % sec_default,
)

args = vars(parser.parse_args())

if args["verbose"]:
pass # do nothing
# print ("verbosity turned on")
if args["number"]:
print("num=%i" % args["number"])

==== match case statement ====
(new in python 3.10)
match args:
case {"sap": "prod", "version": 1}:
...
case {"sap": "prod", "version": 2}:
...
case _: # default case

==File Access==
===Pathlib===
for migrating see table [https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module]
from pathlib import Path

p = Path("dir/test.txt")

log_file = Path(__file__).with_suffix(".log") # __file__ is name of python script

my_fname = p.name # alternative to basename
my_ext = p.suffix
(filepath_without_ext, file_ext) = (p.stem, p.suffix)
my_dir = p.parent
my_parent_dir = p.parents[1]
p2 = p.with_suffix(".json")
p3 = p.parent / (p.name + "-autofix.tex")

# checks
Path(my_file_str).exists()
Path(my_file_str).is_file()
Path(my_file_str).is_dir()

# loop over glob of files matching wildcard
for file_out in Path("mydir").glob("*-autofix.tex"):

# loop over dirs
p = Path() # cwd
list_of_repos = [x for x in p.iterdir() if x.is_dir() and (x / ".git").is_dir()]

alternative using old os functions
Split path into folder, filename, ext
import os
(dirName, fileName) = os.path.split(f)
(fileBaseName, fileExtension) = os.path.splitext(fileName)
fileOut = os.path.splitext(fileIn)[0] + "-out.txt"

===File Manipulations: copy, move, delete, touch===
from pathlib import Path

# copy
from shutil import copyfile
copyfile(Path("file-source.txt"), Path("dir") / Path("file-target.txt"))

# move / rename
Path("file.txt").replace(Path("file2.txt"))

# delete
Path("file.txt").unlink()

# touch
Path("file.txt").touch()

===File size and timestamp===
# size
Path("file.txt").stat().st_size

# timestamp last modified
Path("file.txt").stat().st_mtime

===Dir create/mkdir and delete===
# create dir
from pathlib import Path
Path("myDir1/myDir2").mkdir(parents=True, exist_ok=True)

# delete dir
import shutil
shutil.rmtree(d)

===Loop over Directories===
====Fetch Dir Contents / Loop over Files====
glob via pathlib
for fileOut in Path("mydir").glob("*-autofix.tex"):

Get list of files in directory, filter dirs from list, filter by ext
dir= "/path/to/some/dir"
listoffiles = [ f for f in os.listdir(dir) if os.path.isfile(os.path.join(dir ,f)) and f.lower()[-4:] == ".gpx"]
listoffiles.sort()

====Traverse in Subdirs====
# walk into path an fetch all files matching extension jpe?g
files = []
for (dirpath, dirnames, filenames) in os.walk("."):
dirpath = dirpath.replace("\\", "/")
for file in filenames:
if file.endswith(".txt"):
files.append(dirpath + "/" + file)
elif re.search(r"\.jpe?g$", file, re.IGNORECASE):
files.append(dirpath + "/" + file)

==File Parsing==
===File General===
====File Read====
path_file_in = Path("file.txt")

# via pathlib
cont = path_file_in.read_text(encoding="utf-8") # note: utf-8-sig for UTF-8-BOM

# general
with path_file_in.open(encoding="utf-8") as fh:
cont = fh.read()
# or
lst = fh.readlines()
# or
line = fh.readline()
# or
for line in fh:
print(line)

fh = path_file_in.open(encoding="utf-8")
...
fh.close()

====File Write====
path_file_out = Path("out/1/out.txt")

# write via pathlib
path_file_out.write_text("some text")

# write general
with path_file_out.open(mode="w", encoding="utf-8", newline="\n") as fh:
# w = overWrite file ; a = append to file
# If running Python in Windows, "\n" is automatically replaced by "\r\n".
# To prevent this use newline='\n'
fh.writelines(lst) # no linebreaks
# or
fh.write("\n".join(lst))
# or
for line in lst:
fh.write(line)

# Force update of file contents without closing it
fh.flush()

# alternative
fh = path_file_out.open(mode="w", encoding="utf-8", newline="\n")
...
fh.close()

===CSV===
====CSV Read====
import csv
from pathlib import Path

with Path("data.csv").open(mode="r", encoding="utf-8") as fh: # for Excel use ANSI
csv_reader = csv.DictReader(fh, dialect="excel", delimiter="\t")
for row in csv_reader:
print(f'\t{row["name"]} works in the {row["department"]} department')

====CSV Write====
import csv
from pathlib import Path

# simple
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.writer(fh, delimiter="\t")
csvwriter.writerow(("Date", "Confirmed"))

# dictwriter
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.DictWriter(
fh,
delimiter="\t",
extrasaction="ignore",
fieldnames=["date", "occupied_percent", "occupied", "total"],
)
csvwriter.writeheader()
for d in my_list_of_dicts:
d["occupied_percent"] = round(100 * d["occupied"] / d["total"], 1)
csvwriter.writerow(d)

===JSON===
====JSON Read====
import json
# from file
with path_to_download_file.open(encoding="utf-8") as fh:
d_json = json.load(fh)
# from string
d_json = json.loads(response_text)

def json_read(file_path: Path) -> list[dict[str, str]]:
"""
Read JSON data from file.
"""
with file_path.open(encoding="utf-8") as fh:
json_data = json.load(fh)
return json_data

====JSON Write====
Write dict to file in JSON format, keeping utf-8 encoding
import json

with path_to_download_file.open(mode="w", encoding="utf-8", newline="\n") as fh:
json.dump(my_dict, fh, ensure_ascii=False, sort_keys=False, indent=2)

def json_write(file_path: Path, json_data: list[dict[str, str]]) -> None:
"""
Write JSON data to file.
"""
with file_path.open("w", encoding="utf-8", newline="\n") as fh:
json.dump(json_data, fh, ensure_ascii=False, sort_keys=False, indent=2)

===Excel===
====Excel Read====
import openpyxl
# or xlsxwriter for write-only

workbook = openpyxl.load_workbook(
pathToMyExcelFile,
data_only=True, # read values instead of formulas
read_only=True, # suppresses: "UserWarning: wmf image format is not supported so the image is being dropped"
)
sheet = workbook["mySheetName"]
# or fetch active sheet
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=34, column=1) # index start here with 1
print(cell.value)
# or
print(sheet.cell(column=col, row=row).value)

====Excel Write====
import openpyxl

workbook = openpyxl.Workbook()
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=i, column=j) # index starts at 1
cell.value = "asdf"
workbook.save("out.xlsx")

===Word===
====Word Read====
from io import BytesIO
from pathlib import Path
import pandas as pd
from docx import Document
from docx.document import Document as DocxDocument

def read_doc_to_df(p_doc: Path | BytesIO) -> pd.DataFrame: # noqa: C901, PLR0912
"""Read Word document to DataFrame."""
if isinstance(p_doc, Path):
doc = Document(str(p_doc))
elif isinstance(p_doc, BytesIO):
doc = Document(p_doc)

# list of multiple paragraphs per key
data: dict[str, dict[str, list[str]]] = {}

sections1: list[str] = []
section1 = ""
key = ""
section2 = ""
score = ""

for paragraph in doc.paragraphs:
assert paragraph.style is not None
text = paragraph.text.strip()
if paragraph.style.name == "Title": # doc title
continue
if paragraph.style.name == "Heading 1": # sheet name
section1 = text
if section1 not in sections1:
sections1.append(section1)
key = ""
section2 = ""
elif paragraph.style.name == "Heading 2": # Key of the question
key = text.split(" | ")[0]
key = section1 + "|" + key
section2 = ""
elif paragraph.style.name == "Normal" and text.startswith("[") and "]" in text:
section2, t2 = text[1:].split("]", maxsplit=1)
if section2 == "Score":
score = t2.strip()
data[key][section2] = [score]
elif paragraph.style.name == "Normal":
text = paragraph.text.strip()
# skip requirements leftovers and empty paragraphs
if (section2 == "" and text.startswith("...")) or text == "":
continue
lst = data.setdefault(key, {}).setdefault(section2, [])
lst.append(text)
data[key][section2] = lst
else:
msg = f"Unexpected style: {paragraph.style.name} in paragraph: {text}"
raise ValueError(msg)

# flatten the text from list to str
data2: dict[str, dict[str, str]] = {}
for key, sections in data.items():
for section2, lst in sections.items():
s = "\n".join(lst)
if key not in data2:
data2[key] = {}
data2[key][section2] = s.strip()

df1 = pd.DataFrame.from_dict(data2, orient="index")
df1.index.name = "Key"
df1 = df1.reset_index()
df1[ [ "Sheet", "Key" ] ] = df1["Key"].str.split("|", n=1, expand=True)

return df1

====Word Write====
from io import BytesIO
from pathlib import Path
import pandas as pd from docx import Document
from docx.document import Document as DocxDocument
from docx.shared import Cm

LAYOUT_SMALL_MARGINS = True
if LAYOUT_SMALL_MARGINS:
sections = doc.sections
for section in sections:
section.top_margin = Cm(1)
section.bottom_margin = Cm(1)
section.left_margin = Cm(1)
section.right_margin = Cm(1)

doc.add_heading("My Title", level=0)
doc.add_heading("My Section", level=1)
doc.add_paragraph("Some Text)
p = doc.add_paragraph()
runner = p.add_run(bold text")
runner.bold = True # runner.italic = True

==Regular Expressions==
See [http://docs.python.org/library/re.html]

See [https://pythex.org] for an online tester

multiple flags are joined via pipe |
s = re.sub("asdf.*", r"qwertz", s, flags=re.DOTALL | re.IGNORECASE)

====Lookahead and Lookbehind====
pos lookahead: (?=...)
neg lookahead: (?!...)
pos lookbehind (?<=...)
neg lookbehind (?<!...)

====matching====
import re

# V0: simple 1
myPattern = "(/\*\*\* 0097_210000_0192539580000_2898977_0050 \*\*\*/.*?)($|/\*\*\*)"
myRegExp = re.compile(myPattern, re.DOTALL)
myMatch = myRegExp.search(cont)
assert myMatch != None, f"golden file not found in file {filename}"
cont_golden = myMatch.group(1)

# V1: simple 2
assert (
re.match("^[a-z]{2}$", d_settings["country"]) != None
), f'Error: county must be 2 digit lower case. We got: {d_settings["country"]}'

# V2: find and count
was = r"""Part: ([^<+])"""
cnt_parts = len(re.findall(was, cont))
cont = re.sub(was, r"\n\nTeil: \1\n\n", cont)
assert cnt_parts == 4, f"{cnt_parts} == 4"
assert "Part:" not in cont

=====Match email=====
def checkValidEMail(email: str) -> bool:
# from https://stackoverflow.com/posts/719543/timeline bottom edit
if not re.fullmatch(r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$", email):
print("Error: invalid email")
quit()
return True

====Find all====
myMatches = re.findall('href="([^"]+)"', cont)
for myMatch in myMatches:
print(myMatch)

====substring====
import re

# simple via search
lk_id = re.search('^.*timeseries\-(\d+)\.json$', f).group(1)

# simple via sub
myPattern = "^.*" + s1 + "(.*)" + s2 + ".*$"
out = re.sub(myPattern, r"\1", s)

# more robust including an assert
def substr_between(s: str, s1: str, s2: str) -> str:
"""
returns substring of s, found between strings s1 and s2
s1 and s2 can be regular expressions
"""
myPattern = s1 + '(.*)' + s2
myRegExp = re.compile(myPattern)
myMatches = myRegExp.search(s)
assert myMatches != None, f"E: can't find '{s1}'...'{s2}' in '{s}'"
out = myMatches.group(1)
return out

matchObj = re.search(r"(\d+\.\d+)", text
if matchObj:
price = float( '%s' % (matchObj).group(0) )

====Naming of match groups====
(?P<name>...), see [https://docs.python.org/3/library/re.html]

====Search and Replace====
From [http://www.regular-expressions.info/python.html] 
re.sub(regex, replacement, str) performs a search-and-replace across subject, replacing all matches of regex in str with replacement. The result is returned by the sub() function. The str string you pass is not modified.

s = re.sub(" +", " ", s)

====Splitting====
From [http://docs.python.org/library/re.html] 
split() splits a string into a list delimited by the passed pattern. The method is invaluable for converting textual data into data structures that can be easily read and modified by Python as demonstrated in the following example that creates a phonebook.

First, here is the input. Normally it may come from a file, here we are using triple-quoted string syntax:
>>> input = """Ross McFluff: 834.345.1254 155 Elm Street
...
... Ronald Heathmore: 892.345.3428 436 Finley Avenue
... Frank Burger: 925.541.7625 662 South Dogwood Way
...
...
... Heather Albrecht: 548.326.4584 919 Park Place"""

The entries are separated by one or more newlines. Now we convert the string into a list with each nonempty line having its own entry:
>>> entries = re.split("\n+", input)
>>> entries
['Ross McFluff: 834.345.1254 155 Elm Street',
'Ronald Heathmore: 892.345.3428 436 Finley Avenue',
'Frank Burger: 925.541.7625 662 South Dogwood Way',
'Heather Albrecht: 548.326.4584 919 Park Place']

Finally, split each entry into a list with first name, last name, telephone number, and address. We use the maxsplit parameter of split() because the address has spaces, our splitting pattern, in it:
>>> [re.split(":? ", entry, 3) for entry in entries]
[['Ross', 'McFluff', '834.345.1254', '155 Elm Street'],
['Ronald', 'Heathmore', '892.345.3428', '436 Finley Avenue'],
['Frank', 'Burger', '925.541.7625', '662 South Dogwood Way'],
['Heather', 'Albrecht', '548.326.4584', '919 Park Place']]

==== replace cont by linebreaks====
def replace_cont_by_linebreaks(s: str, regex: str) -> str:
"""
Replace regex in s by the number of linebreaks it originally contained.
"""
myMatches = re.findall(regex, s, flags=re.DOTALL)
for match in myMatches:
linebreaks = match.count("\n")
s = s.replace(match, "\n" * linebreaks, 1)
return s

==Image/Picture/Photo==
from PIL import Image, ImageFilter # pip install Pillow

fileIn = "2018-02-09 13.56.25.jpg"
# Read image
img = Image.open(fileIn)
PROBLEM: 
PIL Image.save() drops the IPTC data like tags, keywords, copywrite, ... 
better using https://imagemagick.org instead when tags shall be kept

====Resize====
# Resize keeping aspect ration -> img.thumbnail
# drops exif data, exif can be added from source file via exif= in save, see below
size = 1920, 1920
img.thumbnail(size, Image.ANTIALIAS)

====Export file====
fileOut = os.path.splitext(fileIn)[0] + "-edit.jpg"
try:
img = Image.open(fileIn)
img.save(fp=fileOut, format="JPEG", quality='keep') # exif=dict_exif_bytes
# JPEG Parameters
# * qualitiy : 'keep' or 1 (worst) to 95 (best), default = 75. Values above 95 should be avoided.
# * dpi : tuple of integers representing the pixel density, (x,y)
except IOError:
print("cannot write file '%s'" % fileOut)

====Export Progressive / web optimized JPEG====
from PIL import ImageFile # for MAXBLOCK for progressive export
fileOut = os.path.splitext(fileIn)[0] + "-progressive.jpg"
try:
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)
except IOError:
ImageFile.MAXBLOCK = img.size[0] * img.size[1]
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)

====JPEG Meta Data: EXIF and IPTC====
=====IPTC: Tags/Keywords=====
from iptcinfo3 import IPTCInfo # this works in pyhton 3!
iptc = IPTCInfo(fileIn)
if len(iptc['keywords']) > 0: # or supplementalCategories or contacts
print('====> Keywords')
for key in sorted(iptc['keywords']):
s = key.decode('ascii') # decode binary strings
print(s)

=====EXIF via piexif=====
import piexif # pip install piexif
exif_dict = piexif.load(img.info['exif'])
print(exif_dict['GPS'][piexif.GPSIFD.GPSAltitude])
# returns list of 2 integers: value and donator -> v / d
# (340000, 1000) => 340m
# (51, 2) => 25.5m

# Modify altitude
exif_dict['GPS'][piexif.GPSIFD.GPSAltitude] = (140, 1) # 140m

# write to file
exif_bytes = piexif.dump(exif_dict)
fileOut = os.path.splitext(fileIn)[0] + "-modExif.jpg"
try:
img.save(fp=fileOut, format="jpeg", exif=exif_bytes, quality='keep')
except IOError:
print("cannot write file '%s'" % fileOut)
or
exif_dict = piexif.load(fileIn)
for ifd in ("0th", "Exif", "GPS", "1st"):
print("===" + ifd)
for tag in exif_dict[ifd]:
print(piexif.TAGS[ifd][tag]["name"], "\t",
tag, "\t", exif_dict[ifd][tag])
print(exif_dict['0th'][306]) # 306 = DateTime

=====EXIF via exifread=====
# Open image file for reading (binary mode)
fh = open(fileIn, "rb")
# Return Exif tags
exif = exifread.process_file(fh)
fh.close()
# for tag in exif.keys():
# if tag not in ('JPEGThumbnail', 'TIFFThumbnail', 'Filename', 'EXIF MakerNote'):
# print("%s\t%s" % (tag, exif[tag]))
print(exif["Image DateTime"])
print(exif["GPS GPSLatitude"])
print(exif["GPS GPSLongitude"])

=====EXIF GPS via PIL=====
# from https://developer.here.com/blog/getting-started-with-geocoding-exif-image-metadata-in-python3
def get_exif(filename):
image = Image.open(filename)
image.verify()
image.close()
return image._getexif()

def get_labeled_exif(exif):
labeled = {}
for (key, val) in exif.items():
labeled[TAGS.get(key)] = val
return labeled

def get_geotagging(exif):
if not exif:
raise ValueError("No EXIF metadata found")
geotagging = {}
for (idx, tag) in TAGS.items():
if tag == "GPSInfo":
if idx not in exif:
raise ValueError("No EXIF geotagging found")
for (key, val) in GPSTAGS.items():
if key in exif[idx]:
geotagging[val] = exif[idx][key]
return geotagging

def get_decimal_from_dms(dms, ref):
degrees = dms[0][0] / dms[0][1]
minutes = dms[1][0] / dms[1][1] / 60.0
seconds = dms[2][0] / dms[2][1] / 3600.0
if ref in ["S", "W"]:
degrees = -degrees
minutes = -minutes
seconds = -seconds
return round(degrees + minutes + seconds, 5)

def get_coordinates(geotags):
lat = get_decimal_from_dms(geotags["GPSLatitude"], geotags["GPSLatitudeRef"])
lon = get_decimal_from_dms(geotags["GPSLongitude"], geotags["GPSLongitudeRef"])
return (lat, lon)

exif = get_exif(fileIn)
exif_labeled = get_labeled_exif(exif)
print(exif_labeled["DateTime"])

geotags = get_geotagging(exif)
print(get_coordinates(geotags))

====Template Matching / Image Regocnition Using CV2 / OpenCV====
see [[Python - CV2]]

====Optical Character Recognition (OCR) ====
see [[Python - OCR]]

==Templates/Snippets==
===Retry on Exception===
for retry_no in range(MAX_RETRIES):
try:
if retry_no > 0:
logger.warning("retrying %d/%d...", retry_no + 1, MAX_RETRIES)
doit()
except json.JSONDecodeError:
logger.exception("JSONDecodeError")
if retry_no == MAX_RETRIES - 1:
# end of retries
raise

===Diff of 2 files===
import difflib

fh1 = p_file1.open("r", encoding="utf-8")
fh2 = p_file1.open("r", encoding="utf-8")
diff = difflib.ndiff(fh1.readlines(), fh2.readlines())
delta = "".join(line for line in diff if line.startswith(("+ ", "- ")))
print(delta)

===GPX parsing===
import gpxpy
import gpxpy.gpx
# Elevation data by NASA: see lib at https://github.com/tkrajina/srtm.py
fh_gpx_file = open(gpx_file_path, 'r')
gpx = gpxpy.parse(fh_gpx_file)
# Loops for accessing the data
for track in gpx.tracks:
for segment in track.segments:
for point in segment.points:
for waypoint in gpx.waypoints:
for route in gpx.routes:
for point in route.points:
# interesting properties of point / waypoint objects:
point.time
point.latitude
point.longitude
point.source
waypoint.name

===TypeHints===
from typing import Any, Dict, cast
creds = cast(dict[str, str], tomllib.load(f)) # type: ignore
o = cast(Dict[str, Any], tomllib.load(f)) # type: ignore
o["sap"] = cast(Dict[str, str], o["sap"])
o["settings"] = cast(Dict[str, str | int | bool], o["settings"])
o["settings"]["sleep_time"] = cast(int, o["settings"]["sleep_time"])

===Pylance/Pyright===
import tomllib
# shows warning: Import "xyz" could not be resolved
# fix by
import tomllib # pyright: ignore

===asserts function argument validation===
aus [https://bildungsportal.sachsen.de/opal/auth/RepositoryEntry/12518195218/CourseNode/94506982489539?0 Python Kurs von Carsten Knoll]
def eine_funktion(satz, ganzzahl, zahl2, liste):
if not type(satz) == str:
print "Datentpyfehler: satz"
return -1
if not isinstance(ganzzahl, int):
print "Datentpyfehler: ganzzahl"
return -2
if not isinstance(liste, (tuple, list)):
print "Datentpyfehler: liste"
return -3
# Kompakteste Variante (empfohlen):
assert zahl2 > 0, "Error: zahl2 ist nicht > 0" # Assertation-Error bei Nichterfuellung

def F(x):
if not isinstance(x, (float, int)):
msg = "Zahl erwartet, %s bekommen" % type(x)
raise ValueError(msg)
return x**2
better:
def F(x):
assert isinstance(x, (float, int)), "Error: x is not of type float or int"
return x**2

assert variant in [
"normal",
"gray",
"cannyedge",
], "Error: variant is not in 'normal', 'gray', 'cannyedge'"

===Exceptions===
see [https://medium.com/@saadjamilakhtar/5-best-practices-for-python-exception-handling-5e54b876a20]

====Catching====
Catch keyboard interrupt and do a "save exit"
try:
i = 0
while 1:
i += 1
print(i)
except KeyboardInterrupt:
print("stopped")

Catch '''all''' exceptions
try:
[...]
except Exception as e:
print("Exception: ", e)

====Raising Exceptions====
if resp.status_code != 200: # noqa: PLR2004
msg = f"Bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise ValueError(msg) from None

Custom Exceptions
try:
raise Exception("HiHo")

raise ValueError¶

===perl grep and map===
from [https://stackoverflow.com/questions/12845288/grep-on-elements-of-a-list]
def grep(list, pattern):
expr = re.compile(pattern)
return [elem for elem in list if expr.match(elem)]
or
filteredList = filter(lambda x: x < 7 and x > 2, unfilteredList)

def map(list, was, womit):
return list(map(lambda i: re.sub(was, womit, i), list))
# was = '.*"(\d+)".*'
# womit = r"\1"

===unit testing using pytest===
install via
pip install pytest
activate in vscode, see [https://code.visualstudio.com/docs/python/testing#_enable-a-test-framework]: To enable testing, use the Python: Configure Tests command on the Command Palette.

see https://docs.pytest.org/en/6.2.x/assert.html#assert

test_1.py:
import myLib # my custom lib to test

class TestClass:
def test_one(self):
x = "this"
assert "h" in x

def test_two(self):
assert myLib.multiply(1, 2) == 2

def test_three(self):
assert myLib.multiply(2, 2) == 2

====project structure====
=====simplest structore: test next to script=====
Alternative without tests dir:
./main.py
./helper.py
./helper_test.py
with helper_test.py:
from helper import fnc1

=====standalone model with src and tests dirs=====
for this dir structure
src/helper.py
tests/helper_test.py
helper_test.py needs
import sys
from pathlib import Path
sys.path.insert(0, (Path(__file__).parent.parent / "src").as_posix())

=====standalone model with tests dir=====
if the test files are placed inside a ./tests/ dir
./main.py
./helper.py
tests/helper_test.py
helper_test.py needs
sys.path.insert(0, Path(__file__).parent.parent.as_posix())
from helper import fnc1

====parametrize====
import pytest
@pytest.mark.parametrize(
("test_input", "expected"),
[("", None), ("PT30M", 30), ("PT3H", 3 * 60), ("PT2H30M", 2 * 60 + 30)],
)
def test_task_est_to_minutes(test_input: str, expected: int) -> None:
assert task_est_to_minutes(test_input) == expected

=====fixures: prepare and cleanup test_data=====
to automatically run before and after each testcase
@pytest.fixture(autouse=True)
def _setup_tests(): # noqa: ANN202
cache_prepare_lists()
cache_prepare_tasks()

yield

cache_cleanup_test_data()

====Test coverage report====
pip install pytest-cov
# console output
pytest --cov

# html report in coverage_report/index.html with details per file
pytest --cov --cov-report=html:coverage_report
to exclude a code block from coverage report, add
# pragma: no cover

=== Sleep / Wait for input ===
sleep for a while
import time
time.sleep(60)

wait for user input
input("press Enter to close")

===Suppress Warnings===
see [https://docs.python.org/3/library/warnings.html#temporarily-suppressing-warnings]
import warnings
warnings.filterwarnings("ignore", message=".*native_field_num.*not found in message.*")

===Logging===
====V4 minimal stdout logging====
import logging
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s")
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
logging.getLogger("httpx").setLevel(logging.WARNING)

# usage of variables and rounding of numbers. (%d also rounds floats)
LOGGER.info("file %s has %d lines with avg %.1f char/line", filename, lines, c_p_l)

====V3 simple logging====
# main file:
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(name)s\t%(message)s",
handlers=[
RotatingFileHandler(
Path(__file__).with_suffix(".log"),
maxBytes=10485760, # 10 MB = 10*1024*1024,
backupCount=1,
encoding="utf-8",
)
],
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(Path(__file__).stem)
logger.info("Start")

# other files:
import logging
logger = logging.getLogger(__name__)

====V2: rotating file and STDOUT====
# 1. setup
import logging
from logging.handlers import RotatingFileHandler

logfile = "myApp.log"
maxBytes = 20 * 1024 * 1024
backupCount = 5
loglevel_console = logging.INFO
loglevel_file = logging.DEBUG

# create logger
logger = logging.getLogger("root")
logger.setLevel(loglevel_file)

# console handler
ch = logging.StreamHandler()
ch.setLevel(loglevel_console)

# rotating file handler
# fh = logging.FileHandler(logfile)
fh = RotatingFileHandler(logfile, maxBytes=maxBytes, backupCount=backupCount)
fh.setLevel(loglevel_file)

# create formatter and add it to the handlers
# %(name)s = LoggerName, %(threadName)s = TreadName
formatter = logging.Formatter(
"%(asctime)s - %(levelname)s - %(name)s - %(threadName)s - %(message)s "
)
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(ch)
logger.addHandler(fh)

logger.debug("DebugMe")
logger.info("Starting")
logger.warning("Attention")
logger.error("Something went wrong")
logger.critical("Something seriously went wrong ")

# 2. in other files/modules now use
import logging

logger = logging.getLogger(__name__)
logger.info("text")

...
except Exception as e:
logger.exception("Unhandeled exception")
quit()

====V1: Simple====
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(message)s", # \t%(name)s
filename=Path(__file__).with_suffix(".log"), # uncomment to switch to stdout
filemode="a",
)
logger = logging.getLogger(__name__)
logger.debug("Some text")
logger.info("Some text")
logger.warning("Some text")
logger.error("Some text")
logger.critical("Some text")
try:
...
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise

===Process Bar===
see [https://tqdm.github.io tqdm]
from tqdm import tqdm
for i in tqdm(range(10000)):
....
or
with tqdm(total=total_iterations, desc="Processing") as progress_bar:
...
progress_bar.update(1)

===CGI Web development===
# Print necessary headers.
print("Content-Type: text/html")
print()

# errors and debugging info to browser
import cgitb
cgitb.enable()

Access URL or Form Parameters
# V2 from https://www.tutorialspoint.com/python/python_cgi_programming.htm
import cgi
form = cgi.FieldStorage()
username = form.getvalue('username')
print(username)

# V1
import sys
import urllib.parse
query = os.environ.get('QUERY_STRING')
query = urllib.parse.unquote(query, errors="surrogateescape")
d = dict(qc.split("=") for qc in query.split("&"))
print(d)

====CGI Backend Returning JSONs====
#!/usr/local/bin/python3.6
# -*- coding: utf-8 -*-

import cgi
import json

# Print necessary headers.
print("Content-type: application/json")
print()

def get_form_parameter(para: str) -> str:
"asserts that a given parameter is set and returns its value"
value = form.getvalue(para)
assert value, f"Error: parameter {para} missing"
assert value != "", f"Error: parameter {para} missing"
return value

response = {}
response['status'] = "ok"

try:
action = get_form_parameter("action")
response['action'] = action
if action == "myAction":
...

except Exception as e:
response['status'] = "error"
d = {"type": str(type(e)), "text": str(e)}
response["exception"] = d

finally:
print(json.dumps(response))

==Databases==
===PostgreSQL===
====Improved helper function====
"""Helper functions for DB access."""

import datetime as dt
import logging

import psycopg2
import psycopg2.extras
from creds_db import credentials as creds_db

# Configure logging
logger = logging.getLogger(__name__)

connection: psycopg2.extensions.connection = None # type: ignore
cursor_query: psycopg2.extensions.cursor = None # type: ignore
cursor_write: psycopg2.extensions.cursor = None # type: ignore

def connect() -> (
tuple[
psycopg2.extensions.connection,
psycopg2.extensions.cursor,
psycopg2.extensions.cursor,
]
):
"""Connect to the database."""
try:
connection = psycopg2.connect(**creds_db)
cursor_query = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)
cursor_write = connection.cursor()
logger.info(
"Connected to database %s on host %s",
creds_db["database"],
creds_db["host"],
)
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise
else:
return connection, cursor_query, cursor_write

def disconnect() -> None:
"""Disconnect from the database."""
try:
if cursor_query:
cursor_query.close()
if cursor_write:
cursor_write.close()
if connection:
connection.close()
logger.info("Disconnected from the database")
except psycopg2.DatabaseError:
logger.exception("Error during disconnection: %s")

def exists(url: str, valid_hours: float) -> bool | None:
"""
Check if a record exists.

Returns
-------
None for missing record
True for valid record
False for expired record

"""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=valid_hours)
sql = """
SELECT delete_at > %s AS "valid"
FROM my_table
WHERE url = %s LIMIT 1;
"""
cursor_query.execute(sql, (delete_at, url))
res = cursor_query.fetchone()
return res[0] if res else None

def query1(url: str) -> dict | None:
"""Execute a query."""
sql = """
SELECT *
FROM my_table
WHERE url = %s LIMIT 1
"""
cursor_query.execute(sql, (url,))
return cursor_query.fetchone() # type: ignore

def insert(url: str, response: str, delete_in_hours: float) -> None:
"""Insert a record."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=delete_in_hours)
sql = """
INSERT INTO my_table (url, response, delete_at)
VALUES (%s, %s, %s);
"""
cursor_write.execute(sql, (url, response, delete_at))
connection.commit()

def delete(url: str) -> None:
"""Delete a record."""
sql = """
DELETE FROM my_table
WHERE url = %s;
"""
cursor_write.execute(sql, (url,))
connection.commit()

def delete_expired(in_hours: float) -> None:
"""Delete records that expire in less than in_hours from now."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=in_hours)

sql = """
DELETE FROM my_table
WHERE delete_at < %s;
"""
cursor_write.execute(sql, (delete_at,))
connection.commit()

connection, cursor_query, cursor_write = connect()

====Basics====
import psycopg2
import psycopg2.extras

credentials = {
"host": "localhost",
"port": 5432,
"database": "myDB",
"user": "myUser",
"password": "myPwd",
}
connection = psycopg2.connect(**credentials)
cursor = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)

l_bind_vars = [["A1", "A2"], ["B1", "B2"]]

sql = """
SELECT * FROM myTable
WHERE 1=1
AND status NOT IN ('CLOSED')
AND ColA = %s
AND ColB = %s
ORDER BY created DESC
"""
cursor.execute(sql, l_bind_vars)
d_data = dict(cursor.fetchone())

====export result to csv file====
sql1 = "SELECT * FROM table"
sql2 = "COPY (" + sql1 + ") TO STDOUT WITH CSV HEADER DELIMITER '\t'"
with open("out.csv", "w") as file:
cursor.copy_expert(sql2, file)

====ReadConfigFile to connect to PostgreSQL====
from [http://www.postgresqltutorial.com/postgresql-python/connect/]
database.ini
[postgresql]
host=dbhost
port=5432
database=dbname
user=dbuser
password=dbpass

from configparser import ConfigParser

def config(filename="database.ini", section="postgresql"):
parser = ConfigParser()
parser.read(filename)
# get section, default to postgresql
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception(
"Section {0} not found in the {1} file".format(section, filename)
)
return db

main.py
import psycopg2
from config import config

def connect():
"""Connect to the PostgreSQL database server"""
conn = None
try:
params = config() # read connection parameters
print("Connecting to the PostgreSQL database...")
conn = psycopg2.connect(**params)
cur = conn.cursor() # create a cursor
print("PostgreSQL database version:")
cur.execute("SELECT version()") # execute a statement
db_version = cur.fetchone()
print(db_version)
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
print("Database connection closed.")

if __name__ == "__main__":
connect()

===SQL Lite / SQLite===
see page [[SQLite]]

===InfluxDB===
see [[InfluxDB]]

==Internet Access==
===Send E-Mails===
see [[Python - eMail]]

===Download data using browser UA / REST===
import requests

headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0",
}
resp = requests.get(
url,
headers=headers,
timeout=3, # timeout in sec, requests should always have a timeout!
# timeout=(1,3), # 1s connect-timout, 3s read-timeout
)
if resp.status_code != 200: # noqa: PLR2004
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download only if cache is too old====
import time
from pathlib import Path
import requests

def fetch_url_or_cache(file_cache: Path, url) -> str:
"""Fetch URL and cache in a file."""
if check_cache_file_available_and_recent(
file_path=file_cache, max_age=3600, verbose=False
):
with file_cache.open(mode="r", encoding="utf-8") as fh:
s = fh.read()
else:
s = fetch(url=url)
with file_cache.open(mode="w", encoding="utf-8", newline="\n") as fh:
fh.writelines(s)
return s

def check_cache_file_available_and_recent(
file_path: Path,
max_age: int = 3500,
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = False
if file_path.exists() and (time.time() - file_path.stat().st_mtime < max_age):
cache_good = True
return cache_good

# or
def check_cache_file_available_and_recent_verbose(
file_path: Path, max_age: int = 3600, *, verbose: bool = False
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = True
if not file_path.exists():
if verbose:
print(f"No Cache available: {file_path}")
cache_good = False
if cache_good and time.time() - (time.time() - file_path.stat().st_mtime < max_age):
if verbose:
print(f"Cache too old: {file_path}")
cache_good = False
return cache_good

def fetch(url) -> str:
"""Fetch URL."""
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0", # noqa: E501
}
resp = requests.get(url, headers=headers, timeout=5)
if resp.status_code == 200: # noqa: PLR2004
return resp.content.decode("ascii")
else: # noqa: RET505
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download using urllib.request====
from urllib.request import urlopen

url = "https://pomber.github.io/covid19/timeseries.json"
with urlopen(url) as response: # noqa: S310
response_text = response.read()

===HTML parsing and extracting of elements===
V2: via BeautifulSoup
from bs4 import BeautifulSoup # pip install beautifulsoup4

soup = BeautifulSoup(cont, features="html.parser")
my_element = soup.find("div", {"class": "user-formatted-inner"})
my_body = my_element.prettify()
# my_body = my_element.encode()
# my_body = str(my_element)

V1: via lxml and xpath
from lxml import html
import requests

page = requests.get(url)
tree = html.fromstring(page.content)
tbody_trs = tree.xpath("//*/tbody/tr")
l_rows = []
for tr in tbody_trs:
l_columns = []
if len(tr) != 15:
continue
for td in tr:
l_columns.append(td.text_content())
l_rows.append(list(l_columns))

===HTML entities to unicode===
import html
cont = html.unescape(cont)

==GUI Interactions==
===Take Screenshot===
import pyautogui # (c:\Python\Scripts\)pip install pyautogui
# pyautogui does only support screenshots on monitor #1
...
screenshot = pyautogui.screenshot()
# screenshot = pyautogui.screenshot(region=(screenshotX,screenshotY, screenshotW, screenshotH))
screenshot = np.array(screenshot)
# Convert RGB to BGR
screenshot = screenshot[:, :, ::-1].copy()

===Mouse Actions===
def clickIt(x,y,key="") :
x0, y0 = pyautogui.position()
if key != "": # crtl, shift
pyautogui.keyDown(key)
pyautogui.moveTo(x, y, duration=0.2)
pyautogui.click(x=x , y=y, button='left', clicks=1, interval=0.1)
if key != "": # crtl, shift
pyautogui.keyUp(key)
pyautogui.moveTo(x0, y0)

===Web Automation===
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time
import glob

class StravaUserMapDL():
def __init__(self):
self.driver = webdriver.Firefox()

def login(self):
driver = self.driver
url = "https://www.somewebpage.com"
email = "myemail"
password = "mypassword"
driver.get(url)

title = driver.title
urlIs = driver.current_url
cont = driver.page_source # as string
FILE = open(filename,"w") # w = overWrite file ; a = append to file
FILE.write(cont)
FILE.close()

# handle login if urlIs != url
if (urlIs != url):
# activate checkbox 'remember_me'
elem = driver.find_element_by_id('remember_me')
if (elem.is_selected() == False):
elem.click()
assert elem.is_selected() == True
elem = driver.find_element_by_id('email')
elem.send_keys(email)
elem = driver.find_element_by_id('password')
elem.send_keys(password)
elem.send_keys(Keys.RETURN)
# Wait until login pages is replaced by real page
urlIs = driver.current_url
while (urlIs != url):
time.sleep(1)
urlIs = driver.current_url
print (urlIs)

# results = driver.find_elements_by_class_name('following')
# results = driver.find_elements_by_tag_name('li')

# print(results[0].text)
assert (urlIs == url)

===Unit Tests using Web Automation===
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time

class PythonOrgSearch(unittest.TestCase):
# def __init__(self,asdf):
# self.driver = webdriver.Firefox()

def setUp(self):
print ("setUp")
# headless mode:
# opts = Options()
# opts.set_headless()
# assert opts.headless # Operating in headless mode
# self.driver = webdriver.Firefox(options=opts)

self.driver = webdriver.Firefox()

def test_search_in_python_org(self):
driver = self.driver
driver.get("http://www.python.org")
self.assertIn("Python", driver.title)
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
print ("fertig: python_org")

def tearDown(self):
print ("tearDown")
print ("close Firefox")
self.driver.close() # close tab
self.driver.quit() # quit browser
# os._exit(1) # exit unittest without Exception

if __name__ == "__main__":
try:
unittest.main()
except SystemExit as e:
os._exit(1)

==Cryptography and Hashing==
====Hashing via SHA256====
def gen_SHA256_string(s: str) -> str:
m = hashlib.sha256()
m.update(s.encode("ascii"))
return m.hexdigest()

====Hashing via MD5====
(MD5 is not secure, better use SHA256)
def gen_MD5_string(s: str) -> str:
m = hashlib.md5()
m.update(s.encode("ascii"))
return m.hexdigest()

====Password hashing via bcrypt====
import bcrypt
pwd = 'geheim'
pwd = pwd.encode("utf-8")
# or
pwd = b'geheim'

hashed = bcrypt.hashpw(pwd, bcrypt.gensalt())
if bcrypt.checkpw(pwd, hashed):
print("It Matches!")
print(hashed.decode("utf-8"))

To use version 2a instead of 2b (default):
bcrypt.gensalt(prefix=b"2a")

==Multiprocessing, subprocesses and Threading==
see [[Python_-_Multithreading]] as well

use processes for CPU limited work 
use threads for I/O limited work

===Simple single process===
import subprocess
process = subprocess.run(["sudo", "du", "--max-depth=1", mydir], capture_output=True, text=True)
print (process.stdout)
old, depricated way:
os.system( "gnuplot " + gpfile)

===Multiprocessing===
see [[Python - Multithreading]] as well

V2 using pool and starmap
import multiprocessing
import os

def worker(i: int, s: str) -> list:
result = (i, s, os.getpid())
return result

if __name__ == "__main__":
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n" + str(i))
l_pile_of_work.append((tup))
# gen pool of processes
num_processes = min(multiprocessing.cpu_count(), len(l_pile_of_work))
pool = multiprocessing.Pool(processes=num_processes)
# start processes on pile of work
l_results_unsorted = pool.starmap(
func=worker, iterable=l_pile_of_work # each item is a list of 2 parameters
)

# or if only one parameter:
# l_results_unsorted = pool.map(doit_de_district, l_pile_of_work)

l_results = sorted(l_results_unsorted) # sort by i

V1
import subprocess
l_subprocesses = [] # queue list of subprocesses
max_processes = 4

def process_enqueue(new_process_parameters):
global l_subprocesses
# wait for free slot
while len(l_subprocesses) >= max_processes:
process_remove_finished_from_queue()
time.sleep(0.1) # sleep 0.1s
process = subprocess.Popen(new_process_parameters,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
l_subprocesses.append(process)

def process_remove_finished_from_queue():
global l_subprocesses
i = 0
while i <= len(l_subprocesses) - 1:
process = l_subprocesses[i]
if process.poll != None: # has already finished
process_print_output(process)
l_subprocesses.pop(i)
else: # still running
i += 1

def process_print_output(process):
"""waits for process to finish and prints process output"""
stdout, stderr = process.communicate()
if stdout != '':
print(f'Out: {stdout}')
if stderr != '':
print(f'ERROR: {stderr}')

def process_wait_for_all_finished():
global l_subprocesses
for process in l_subprocesses:
process_print_output(process)
l_subprocesses = [] # empty list of done subprocesses

process_enqueue(l_parameters1)
...
process_enqueue(l_parameters999)
process_wait_for_all_finished()

===Threading===
import threading
import queue
import os
import time

def worker(q_work: queue.Queue, results: dict):
while not q_work.empty():
i, s = q_work.get()
time.sleep(.1)
result = (i, s, os.getpid())
results[i] = result
q_work.task_done()

if __name__ == '__main__':
d_results = {} # threads can write into dict
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n"+str(i))
l_pile_of_work.append((tup))
# convert list of work to queue
q_pile_of_work = queue.Queue(
maxsize=len(l_pile_of_work)) # maxsize=0 -> unlimited
for params in l_pile_of_work:
q_pile_of_work.put(params)
# gen threads
num_threads = 100
l_threads = [] # List of threads, not used here
for i in range(num_threads):
t = threading.Thread(name='myThread-'+str(i),
target=worker,
args=(q_pile_of_work, d_results),
daemon=True)
l_threads.append(t)
t.start()
q_pile_of_work.join() # wait for all threas to complete
l_results_unsorted = d_results.values()
l_results = sorted(l_results_unsorted) # sort by i

===asyncio — Asynchronous I/O===
see https://docs.python.org/3/library/asyncio-task.html
import asyncio
import time

# basics
# task = asyncio.create_task(coro())
# Wrap the coro coroutine into a Task and schedule its execution. Return the Task object.

# sleep
# await asyncio.sleep(1)

# Running Tasks Concurrently and gathers the return values in list L
# L = await asyncio.gather(coro(x1,y1), coro(x2,y2), coro(x3,y3))

async def say_after(delay, what):
# Coroutines declared with the async/await syntax
await asyncio.sleep(delay)
print(what)

async def main():
# The asyncio.create_task() function to run coroutines concurrently as asyncio Tasks.
task1 = asyncio.create_task(
say_after(1, 'hello'))

task2 = asyncio.create_task(
say_after(1, 'world'))

print(f"started at {time.strftime('%X')}")

# Wait until both tasks are completed
await task1
await task2

print(f"finished at {time.strftime('%X')}")

asyncio.run(main())

==Pandas==
see [[Pandas]]

==Matplotlib==
see [[Matplotlib]]

== GUI via tkinter==
import tkinter as tk # no need to install via pip

class App(tk.Tk):
def __init__(self):
super().__init__()
self.title("robot-CClicker")
self.geometry("200x200+0+1080")
self.resizable(width=False, height=False)

self.l_buttons = []
self.__create_widgets()

def __create_widgets(self):
self.btn_click500 = tk.Button(
master=self,
text="500 clicks",
width=20,
command=lambda: self.clickBigCookie(500),
)
self.l_buttons.append(self.btn_click500)

for button in self.l_buttons:
button.pack(anchor=tk.W)

def clickBigCookie(self, num):
# TODO: the disabling of the buttons is not working
for button in self.l_buttons:
button["state"] = tk.DISABLED
helper.clickIt(self.posBigCockie[0], self.posBigCockie[1], num=num)
for button in self.l_buttons:
button["state"] = tk.NORMAL

if __name__ == "__main__":
app = App()
app.mainloop()

==Protobuf==
===convert .proto file to python class===
see https://www.datascienceblog.net/post/programming/essential-protobuf-guide-python/
protoc my_message.proto --python_out ./

===read/decode protobuf message===
parse message from file
import machine_message_pb2

with open("out.bin", "rb") as f:
my_message = my_message_pb2.my_message()
my_message.ParseFromString(f.read())
print(machine_message)

===create/encode protobuf message===
import machine_message_pb2

my_message = my_message_pb2.my_message()
my_message.data.field1 = 1
my_message.data.field2 = "asdf"

with open("out.bin", "wb") as f:
f.write(my_message.data.SerializeToString())

==venv / virtual environments==
pip install --upgrade pip
python -m venv .venv --prompt $(basename $(pwd))
source .venv/bin/activate
pip install -r requirements.txt
...
deactivate

==Pydantic data validation==
===Function input parameter validation===
see https://docs.pydantic.dev/usage/validation_decorator/
@validate_arguments
def sum(x: int, y: float) -> float:
print(x, y)
return x + y

print(sum(1.1, 1.1))
# -> 2.1

==Read config file==
===TOML===
see https://learnxinyminutes.com/docs/toml/

minimal example:
config.toml
# SAP API endpoint
[general]
sleep_time = 60
use_proxy = true
tool.py
try:
import tomllib # comes with python3.11
except ModuleNotFoundError:
import tomli as tomllib # pip install tomli
with open("config.toml", "rb") as f:
o = tomllib.load(f) # type: ignore
if o["settings"]["use_proxy"]:
...
for validation, see https://realpython.com/python-toml/

====TOML Write====
In for deployments I sometime want to modify a TOML config file for production, based on the dev version.
import tomli_w # pip install tomli-w
p_in = Path(__file__).parent.parent / ".streamlit/config.toml"
p_out = p_in.parent / "config-prod.toml"
with p_in.open("rb") as fh:
o = tomllib.load(fh)
o["server"]["fileWatcherType"] = "none"
del o["server"]["address"]
with p_out.open("wb") as fh:
tomli_w.dump(o, fh)

===ConfigParser: INI Config File Reading===
better use TOML

Config.ini
[Section1]
Cursor = 205E18
Grandma = 18E18
Farm = 11E18
Mine = 514E18
Factory = 155E18
test.py
from configparser import ConfigParser

config = ConfigParser(
interpolation=None
) # interpolation=None -> treats % in values as char % instead of interpreting it
config.read("Config.ini", encoding="utf-8")

print(config.getint("Section1", "key1"))
print(config.getfloat("Section1", "key2"))
print(config.get("Section1", "key3"))

for sec in config.sections():
d_settings = {}
for key in config.options(sec):
value = config.get(sec, key)
d_settings[key] = value
print("%15s : %s" % (key, value))

==VCard / vcf==
import codecs
import vobject # pip install vobject

obj = vobject.readComponents(codecs.open(file_in, encoding="utf-8").read()) # type: ignore
contacts: list[vobject.base.Component] = list(obj) # type: ignore

card = contacts[0]

if "bday" not in card.contents:
continue

# bday: remove 'VALUE': ['DATE']
card.contents["bday"][0].params = {} # type: ignore

# remove all fields but "bday", "n"
for key in card.contents.copy(): # loop over copy, to allow for deleting keys
if key not in ("n", "bday"):
del card.contents[key]

# recreate fn based on n
card.add("fn")
n = card.contents["n"][0]
fn = f"{n.value.given} {n.value.additional} {n.value.family}" # type: ignore
fn = re.sub(r"\s+", " ", fn)
card.fn.value = fn # type: ignore

with open("out.vcf", mode="w", encoding="utf-8", newline="\n") as fhOut:
fhOut.write(card.serialize())

==Sentry Exception monitoring==
see [[Sentry]]
==MQTT==
#!/usr/bin/env python3.12
"""Read power data from Tasmota device via MQTT."""

import json
from typing import Any

import paho.mqtt.client as mqtt
from mqtt_credentials import hostname, password, port, username
from paho.mqtt.reasoncodes import ReasonCode

def on_connect(
mqtt_client: mqtt.Client,
userdata: Any,
flags: dict[str, int],
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client connects MQTT broker.""" # noqa: D401
if reason_code == 0:
print("Connected to MQTT")
# print(f"MQTT protocol version: {mqtt_client._protocol}")

# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
mqtt_client.message_callback_add("tele/tasmota_MT681/SENSOR", on_message)
mqtt_client.subscribe([("tele/tasmota_MT681/SENSOR", 0)])

else:
print(f"Connection to MQTT broker failed. Error: {reason_code}")

def on_disconnect(
mqtt_client: mqtt.Client,
userdata: Any,
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client is disconnected from the MQTT broker.""" # noqa: D401
if reason_code > 0:
print(f"Unexpected disconnection. Error: {reason_code}")

def on_message(
mqtt_client: mqtt.Client,
userdata: Any,
message: mqtt.MQTTMessage,
) -> None:
"""Callback when a Tasmota message is received from the MQTT broker.""" # noqa: D401
s = message.payload.decode()
data = json.loads(s)
# {'Time': '2024-03-16T06:44:08', 'MT681': {'Total_in': 16.4396, 'Power_cur': 80, 'Total_out': 14.6403}} # noqa: E501

total_i = data["MT681"]["Total_in"]
total_o = data["MT681"]["Total_out"]

print(f"In:\t{total_i}\nOut:\t{total_o}")

mqtt_client.disconnect()
mqtt_client.loop_stop()

try:
# Create an MQTT client instance
mqtt_client: mqtt.Client = mqtt.Client(
mqtt.CallbackAPIVersion.VERSION2, "Python-Client-1"
) # type: ignore
mqtt_client.username_pw_set(username, password)

mqtt_client.on_connect = on_connect

# Connect to the MQTT broker
mqtt_client.connect(hostname, port, keepalive=60)

# NOT: while True, as that eats the CPU !!!
mqtt_client.loop_forever()
except KeyboardInterrupt:
print("KeyboardInterrupt, disconnecting from MQTT broker.")
mqtt_client.disconnect()
mqtt_client.loop_stop()

==Caching Functions==
from functools import lru_cache

@lru_cache(maxsize=128) # None equals to @cache declarator
def fnc(n:int):

==Performance/Resources==
===RAM/Memory consumption/bottleneck===
import tracemalloc
tracemalloc.start()

# ...
# Code
# example:
lst = list(range(1_000_000))
# ...

max_bytes = tracemalloc.get_traced_memory()[0]
print(f"{round(max_bytes / 10**6)}MB")

tracemalloc.stop()

===Profiling / Count and Runtime per Function===
call_stats = {}

def track_function_usage(func: Callable) -> Callable:
"""Annotation for gathering runtime statistics."""

@wraps(func)
def wrapper(*args: tuple, **kwargs: dict): # noqa: ANN202
start_time = time.time()
result = func(*args, **kwargs) # Call the original function
end_time = time.time()
call_stats[func.__name__]["calls"] += 1
call_stats[func.__name__]["total_time"] += end_time - start_time
return result
return wrapper

==JWT Token==
try:
decoded_token = jwt.decode(token, options={"verify_signature": False})
except jwt.ExpiredSignatureError:
msg = "Token has expired."
logger.exception(msg)
except jwt.InvalidTokenError:
msg = "Invalid token."
logger.exception(msg)

==Streamlit==
see https://github.com/entorb/streamlit-examples for a collection of my examples

===Minimum Example===
# src/app.py
from pathlib import Path
import streamlit as st
from streamlit.logger import get_logger
logger = get_logger(Path(__file__).stem)
logger.info("Start")
st.set_page_config(page_title="AppTitle", page_icon=None, layout="wide")
st.title("MyPageTitle")
st.header("MyHeader")
st.subheader("MySubHeader")
sel_param = st.selectbox("Parameter", ("count", "sum"))

# .streamlit/config.toml
[browser]
gatherUsageStats = false

[server]
port = 8501

# run it
streamlit run src/app.py

====Reduce Logging for K8s Deployed Docker Containers====
# reduce log output
for logger_name in [
"streamlit.runtime.caching.cache_utils",
"streamlit.runtime.caching.storage.in_memory_cache_storage_wrapper",
"streamlit.runtime.runtime",
"urllib3.connectionpool",
]:
get_logger(logger_name).setLevel(logging.WARNING)

===Chart via Altair===
====Line Chart====
chart = st.line_chart(df["value"])
# corresponds to
import altair as alt
c = (
alt.Chart(df)
.mark_line()
.encode(
x=alt.X("created", title=None),
y=alt.Y("value", title=None),
)
)
st.altair_chart(c, use_container_width=True)

====Bar Chart====
chart = (
alt.Chart(df)
.mark_bar()
.encode(
x=alt.X("yearmonth(date):T", title="Month"),
y=alt.Y(f"count:Q", title=None),
color="status:N",
tooltip=["yearmonth(date):T", "status:N", "cnt:Q"],
)
# .properties(width=600, height=400)
)
st.altair_chart(chart, use_container_width=True)

:T stands for Temporal. It indicates that the field contains date or time values.
:N stands for Nominal. It indicates that the field contains categorical data, which represents discrete categories or labels.
:Q stands for Quantitative. It indicates that the field contains numerical data that can be measured and compared.

===Excel Download===
def excel_download_buttons(df: pd.DataFrame, file_name: str = "export.xlsx") -> None:
"""Show prepare data and download buttons."""
if st.button(label="Click to prepare download"):
buffer = io.BytesIO()
with pd.ExcelWriter(buffer, engine="xlsxwriter") as writer:
df.to_excel(writer, sheet_name="Sheet1", index=False)
writer.close()

st.download_button(
label="Download data as Excel",
data=buffer,
file_name=file_name,
mime="application/vnd.ms-excel",
)

==UV Package Manager==
# create/update lockfile
uv lock --upgrade
# sync venv
uv sync --upgrade
# extract package info to requirements.txt
uv export --format requirements.txt --no-dev --no-hashes -o requirements.txt

Update Python version for my current project
uv venv --python=3.13.7

===Update UV===
uv self update
# or
brew update && brew upgrade uv

===Update Python Versions===
uv python upgrade

===UV + Docker on readOnlyRootFilesystem===
Example 1: https://github.com/SWE-Group-23/CM22007---Source/blob/10d02004f3a4208f7b30d37a0a87e89532a23575/create-service.sh#L117

Dockerfile
WORKDIR /app
COPY . .

ENV CASS_DRIVER_NO_EXTENSIONS=1

RUN --mount=type=cache,target=/root/.cache/uv \\
--mount=type=bind,source=uv.lock,target=uv.lock \\
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \\
uv sync --frozen --no-install-project

RUN uv pip install -e shared/

RUN addgroup -S group1 && adduser -S user1 -G group1 -u 1000
RUN chown -R user1:group1 /app
USER user1:group1

CMD ["uv", "run", "main.py"]

Deployment
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /home/user1/.cache/uv
name: uv
- mountPath: /tmp
name: temp

Example 2: [https://hynek.me/articles/docker-uv/ Production-ready Python Docker Containers with uv]

Dockerfile (without the comments)
ENV UV_LINK_MODE=copy \
UV_COMPILE_BYTECODE=1 \
UV_PYTHON_DOWNLOADS=never \
UV_PYTHON=python3.12 \
UV_PROJECT_ENVIRONMENT=/app

RUN --mount=type=cache,target=/root/.cache \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync \
--locked \
--no-dev \
--no-install-project

COPY . /src
WORKDIR /src
RUN --mount=type=cache,target=/root/.cache \
uv sync \
--locked \
--no-dev \
--no-editable

=== Vulture: Search for dead code ===
see [https://github.com/jendrikseipp/vulture GitHub]

pyproject.toml
[tool.vulture]
paths = ["src"]
# exclude = ["src/models.py"]
ignore_names = [
"LOGGER",
]

.pre-commit-config.yaml
# Vulture: find dead code
- repo: https://github.com/jendrikseipp/vulture
rev: v2.16
hooks:
- id: vulture

run
pip install vulture
vulture src/

uv add --dev vulture
uv run vulture src/

Python

2026-04-12T12:29:27Z

Torben: /* UV + Docker on readOnlyRootFilesystem */

[[Category:Coding]][[Category:Python]]

==Getting Started==
===Install===
====Python====
Best use [https://docs.astral.sh/uv/ UV] for managing python and package versions.

* for Windows: get and install Python from https://www.python.org
* for MacOS: use [https://github.com/pyenv/pyenv?tab=readme-ov-file#a-getting-pyenv pyenv]
brew install xz
brew install pyenv

echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[ [ -d $PYENV_ROOT/bin ] ] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init - zsh)"' >> ~/.zshrc
exec "$SHELL"

pyenv install --list
pyenv install 3.13.5
pyenv global 3.13.5

vim ~/.zshrc
# add
if command -v pyenv 1>/dev/null 2>&1; then
eval "$(pyenv init -)"
fi

====Editor: Visual Studio Code====
excellent and free source-code editor that supports many languages.
* get and install from https://code.visualstudio.com

See Wickie page [[Visual_Studio_Code]] for general setup, extension, config...

Python Extensions
* Python
* Pylance
* [https://marketplace.visualstudio.com/items?itemName=charliermarsh.ruff Ruff Formater and Linter]
* [https://marketplace.visualstudio.com/items/?itemName=tamasfe.even-better-toml Even Better TOML]
Settings (CTRL + ,)
* Extensions -> Python -> Formatting: Provider = black
* Text Editor -> Editor: Format On Save
* Text Editor -> Files:Eol -> \n
* Linter: Ruff
Settings in settings.json
"python.analysis.completeFunctionParens": true,
"python.analysis.autoImportCompletions": true,
"python.analysis.inlayHints.functionReturnTypes": true,
"python.analysis.typeCheckingMode": "strict",
// Ruff
"[python]": {
"editor.defaultFormatter": "charliermarsh.ruff",
"editor.formatOnSave": true,
"editor.codeActionsOnSave": {
"source.fixAll": "explicit",
"source.organizeImports.ruff": "explicit"
},
},

Run your code
CTRL + F5 : run
F5 : run in debugger

====Code Formatter Ruff====
use software for handling the code formatting like "ruff" or "black", where Ruff is much faster and additionally provides linting.
pip install ruff
than activate in editor like vs code

===My Template===
see latest version at https://github.com/entorb/template-python

"""Main file."""
# by Torben Menke https://entorb.net
import logging
from pathlib import Path

def init_logging() -> None:
"""Initialize logging."""
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(
level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s"
)
logging.getLogger("httpx").setLevel(logging.WARNING)

init_logging()
LOGGER = logging.getLogger(__name__)

def main():
LOGGER.info("Moin Moin")

if __name__ == "__main__":
main()

"""Other file"""
import logging
LOGGER = logging.getLogger(__name__)

===Compile to .exe===
pip install pyinstaller
pyinstaller --onefile --console your.py
# for Excel and Matplotlib these options are required
--hidden-import=openpyxl --hidden-import=matplotlib --hidden-import pandas.plotting._matplotlib
([[Python - py2exe]] is deprecated)

==Basics==
=== Naming Conventions===
[https://google.github.io/styleguide/pyguide.html Google Python Style Guide]:
module_name, package_name, ClassName, method_name, ExceptionName, function_name, GLOBAL_CONSTANT_NAME, global_var_name, instance_var_name, function_parameter_name, local_var_name

====Doc Strings====
It is best practice to start a file with a docstring:
"""My Script"""
This can be accessed linke this:
title = __doc__[:-1] # type: ignore

===Installing packages===
python -m pip install --upgrade pip

pip install somemodule
# or
pip3 install somemodule
# or read from file
pip install -r requirements.txt

# uninstall
pip uninstall somemodule

# using a web proxy
# set proxy for windows cmd session
SET HTTPS_PROXY=http://myProxy:8080
(afterwards --proxy setting below no longer required
or
pip install --proxy http://myProxy:8080 somemodule

# list outdated packages
pip list --outdated

# update package
pip install --upgrade pyinstaller

# updating all via Windows Powershell (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | %{$_.split('==')[0]} | %{pip install --upgrade $_}

# updating all via Bash (from [https://thecesrom.dev/2021/05/26/the-one-liner-for-updating-pip-and-all-outdated-packages/])
pip freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -n1 pip install --upgrade

# downgrade
pip install --upgrade pandas==1.2.4

====Oneline Updater for requirements.txt====
https://pkgui.com/pip

====update packages====
pip install pip-review
pip-review --auto

====generate requirements.txt====
pip install pipreqs
pipreqs ./

===Loops===
ATTENTION: The loops do not create a new variable scope. Only functions and modules introduce a new scope!
for p in all_files:
print(p)

for i, p in enumerate(all_files):
print(i, p)

for i in range(10):
print(i)
# del i

while i <= 100:
i += 1
...
if sth1:
continue # start next loop
if sth2:
break # exit loop

# inline if (requires a dummy else):
print("something") if sth else 0

===Math===
see [[Python - Math]] for linear regression

Modulo
15 % 4
# > 3

Integer Division
17 // 4
# > 4

====Random====
import random
random.randint(1000000, 9999999)

==Basic Objects==
===Variables===
del var # delete / undef a variable
var = None # sets to null

# check if variable is defined
if "var" in locals():
pass
# for object oriented projects:
if "var" in self.__dict__:
pass

# Access global variables in functions
var = 123

def test() -> None:
global var # point to global instead of creation of local var
var = 321

===Strings===
# num <-> str
s = str(i) # int to string
f = float(s) # str -> float
i = int(s)
str(round(f, 1)) # round first
# tests
s.isdigit() # 0-9
# note isdecimal() does also not match '1.1'

# printf: 1 digit
s = f"{value:0.1f}"
s = "%0.1f" % value

====Modify Strings====
# get string from prompt
s = input("Enter Text: ")

s = s.strip() # trim spaces from both sides, rstrip for right only
s = s.lower() # lower case
s = s.upper() # upper case
s = s.title() # upper case for first char of word

# upper case first letter of each word and also removes multiple and trailing spaces
import string
s = string.capwords(s)

# replace
s.replace(x, y)

# trim whitespaces from left and right
s.strip()

# replace all (multiple) whitespaces by single space ' '
s = " ".join(s.split())

# generate key value pairs from dict
# key1=value1&key2=value2
param_str = "&".join("=".join(tup) for tup in dict.items())

# repeat string multiple times
s * 5 # = s+s+s+s+s

====Substrings====
# find a substring:
if x in s: # True / False
pass
if len(s) > 0:
pass

# handling substrings
a = "abcd"
b = a[:1] + "o" + a[2:]
# > 'aocd'

s = "Hello there !bob@"
i1 = s.find("!") + 1
i2 = s.find("@")
substr = s[i1:i2]

def substr_between(s: str, s1: str, s2: str) -> str:
assert s1 in s, f"E: can't find '{s1}' in '{s}'"
assert s2 in s, f"E: can't find '{s1}' in '{s}'"
i1 = s.find(s1) + len(s1)
i2 = s.find(s2)
assert i1 < i2, f"E: '{s1}' not before '{s2}' in '{s}'"
return s[i1:i2]

====Binary, raw strings, html encoding====
# Binary Strings
sb = b"asdf"
# or
sb = str.encode("asdf")
s = sb.decode("ascii") # decode binary strings
sb = s.encode("ascii") # encode string to binary

# raw string
s = r"c:\Windows" # no escape of \ needed

# convert utf-8 to html umlaute
s = "Nürnberg".encode("ascii", "xmlcharrefreplace").decode()
# -> Nürnberg

====Guess encoding via chardet====
import chardet # pip install chardet
result = chardet.detect(raw_data)
if result["encoding"] and result["confidence"] > 0.5: # noqa: PLR2004
encoding = result["encoding"]
else:
encoding = "utf-8"

there is a much faster alternative [https://github.com/PyYoshi/cChardet cchardet], but that requires Microsoft Visual C++ 14.0

====Merge variables in string / sprintf====
print("Renner =", i)
print("Renner = %3d" % i) # leading 0's
print(f"Renner = {i}")

# place formatted numbers in a string / sprintf
s = "The {:.1f}% {} cost {:05.2f} euros".format( 5.1, "beer", 3.50)
print(s)
# > The 5.1% beer cost 03.50 euros

s = f"The length is {72.8958:.2f} meters"
# > The length is 72.90 meters

===Lists===
like @array in Perl
lst = [1, 2, 3, 4, 5, 6]
lst = [x / 2 for x in range(10)]
lst = [None] * 10 # Initiate list of None elements:
len(lst) # length
lst2 = lst[0:10] # get elements 0-10
for i in lst:
print(i)

====clone list====
# dangerous: creates a link to the original list
list_2 = my_list # M's elements are LINKS to L's
# clones can be achieved via:
list_2 = my_list.copy
list_2 = my_list[:]
list_2 = list(my_list)

====list modifications====
lst.pop() # returns and removes the last item
lst.pop(i) # returns and removes the item at position int i
lst.insert(i, x) # insert item x at position int i
lst.remove(x) # removes the first occurrence of item x
lst.append(x) # append a single element
lst.extend(m) # append elements of another list

lst.reverse() # reverse the order of the list
lst = sorted(["B", "a", "c"], key=str.casefold) # case insensitive / ignore case

# list to string
s = "".join(lst)
s = "\n".join(lst)
# string to list
lst = s.split() # default: split on space
lst = s.split(",")

# remove empty values from end
while L[-1] == "":
L.pop()

# check and remove items from list
# from https://stackoverflow.com/a/6024599
# iterates in-situ via reversed index
for i in range(len(lst) - 1, -1, -1):
element = lst[i]
if check(element):
del lst[i]

====check if item is in list====
x in lst
x not in lst
lst.count(x) # how many items x are in the list
i = lst.index("word") # find in list / returns the position of the first match in list

====pair 2 lists====
# zip: merge 2 lists to list of tuples
data = list(zip(data_x, data_y, strict=True))

# unzip: split list of pairs into 2 lists
data_x, data_y = zip(*data, strict=True)

# Cartesian product of lists / tuples
import itertools
for i in itertools.product(*list_of_lists):
print(i)

====sort multidim list====
lst = sorted(lst, key=lambda x: x[0], reverse=False)
lst = sorted(lst, key=lambda row: (row["Wann"], row ["Wer"]), reverse=False)

====filter list====
lines = [x for x in lines if not x.startswith ("word")]
lst = ["asdf", "asdf2", "qwertz"]
lst = [elem for elem in lst if "asdf" in elem]

====for each element====
# trim/strip spaces for each element
lines = [s.strip() for s in lines]
# modify each item in list by adding constant string
l = [s + ';' + v for v in l]

# modify item in list
for idx, line in enumerate(cont):
if "K1001/1" in line:
line = "K1001/1 Test Nr " + str(i) + "\n"
cont[idx] = line
break

===Tuples===
Ordered sequence, with no ability to replace or delete items
tpl = (1,2,3,4,5,6)

list -> tuple
tpl = tuple(lst)

combine 2 tuples
tpl = tpl1 + tpl2

===Dictionaries===
like %hash in Perl
d = {"key1": "value1", "key2": 123}
d["key3"] = (1, 2, 3)
del d["key3"]

len(d)
d.clear()
d.copy()
d.keys()
d.values()
d.items() # returns a list of tuples (key, value)
d.get(k) # returns value of key k
d.get(k, x) # returns value of key k; if k is not in d it returns x
count = d.get("key", 0) + 1 # nice for counters
d.pop(k) # returns and removes item k
d.pop(k, x) # returns and removes item k; if k is not in d it returns x
# checks
x in d
x not in d

# dict can have tuple as key
tpl = ("key1", "key2", 123)
d[tpl] = 123456

# loop over all key-value pairs
for key, value in d.items():
print(f"{key} = {value}")

# sorted keys:
for key in sorted(dict.keys()):
pass
# sorted values, reversed
for key, value in sorted(d.items(), key=lambda item: item[1], reverse=True):
pass

# join / merge 2 dicts
d.update(d2)

# flatten dict of dict
flat = d.copy()
meta = d.pop("metadata")
flat.update(meta)

====MultiDim Dictionaries====
d = {}
d["item1"] = {}
d["item1"]["value"] = 1.909e18
d["item1"]["name"] = "Item 1"
d["item2"] = {}
d["item2"]["value"] = 1.725e18
d["item2"]["name"] = "Item 2"

for k in d.keys():
d[k]["value"] = d[k]["value"] * 1.1

def find_common_strings(dict1: dict, dict2: dict) -> dict[Any, Any]:
"""Find common strings in two dict, returning a new dict of those strings."""

def recursive_match(d1: dict, d2: dict) -> dict[Any, Any]:
common_dict = {}
for key in d1: # noqa: PLC0206
if key in d2:
if isinstance(d1[key], dict) and isinstance(d2[key], dict):
# Recurse if both values are dicts
nested_common = recursive_match(d1[key], d2[key])
if nested_common: # Add to result if common values are found
common_dict[key] = nested_common
elif isinstance(d1[key], str) and isinstance(d2[key], str):
# Check if both are strings and identical
if d1[key] == d2[key]:
common_dict[key] = d1[key]
return common_dict

return recursive_match(dict1, dict2)

def update_dict_with_new_values(original_dict: dict, new_values: dict) -> dict:
"""Update strings in the original dict with new values from the new_values dict."""

def recursive_update(d1: dict, new_vals: dict) -> None:
for key in new_vals: # noqa: PLC0206
if key in d1:
if isinstance(d1[key], dict) and isinstance(new_vals[key], dict):
# Recurse if both values are dicts
recursive_update(d1[key], new_vals[key])
elif isinstance(d1[key], str) and isinstance(new_vals[key], str):
# Update the string
d1[key] = new_vals[key]

recursive_update(original_dict, new_values)
return original_dict

===Datetime, Date and Time===
import datetime as dt
from zoneinfo import ZoneInfo
TZ_DE = ZoneInfo("Europe/Berlin")
TZ_UTC = ZoneInfo("UTC")

====Create====
# date
date = dt.date(2023, 12, 31)
date = dt.date.fromisocalendar(int(year), int(week), int(daynum)) # daynum: 1..7
date = dt.date.fromisoformat("2020-03-10")
date = dt.datetime.strptime(datestr, "%y%m%d").date()
date_today = dt.datetime.now(tz=dt.UTC).date()
date_yesterday = dt.datetime.now(tz=TZ_DE).date() - dt.timedelta(days=1)
days_overdue = (date_completed - date_due).days
# to add one month (which is not supported by timedelta)
from dateutil.relativedelta import relativedelta
date_end = date_start + relativedelta(months=1)
delta_days = 1 + (end - start).days

# datetime
dt_now = dt.datetime.now(tz=TZ_DE)
my_dt = dt.datetime(2023, 12, 31, 14, 31, 56, tzinfo=TZ_DE) # 2023-12-31 14:31:56
my_dt = dt.datetime.fromtimestamp(my_timestamp, tz=TZ_DE)
my_dt = dt.datetime.fromisoformat ("2017-01-01T12:30:59.000000")
my_dt = dt.datetime.fromisoformat("2020-03-10 06:01:01+00:00")
s = "2020-03-10T06:01:01Z"
my_dt = dt.datetime.fromisoformat(s.replace("Z", " +00:00"))
my_date_local = my_dt.astimezone(tz=TZ_DE).date()

# datetime -> date
my_date = my_dt.date()
# date -> datetime
my_date = dt.date(2023, 12, 31)
my_dt = dt.datetime(my_date.year, my_date.month, my_date.day, tzinfo=TZ_DE)

# to string
date_str = my_dt.strftime("%y%m%d")
date_str = my_dt.strftime("%Y-%m-%d %H:%M:%S")
# alternative using time
date_str = time.strftime("%Y-%m-%d %H:%M:%S")
# now in UTC without milliseconds
date_str = dt.datetime.now(tz=dt.timezone.utc).replace(microsecond=0).isoformat() + "Z"

# remove timezone
my_dt = my_dt.replace(tzinfo=None)

# German format
import locale
locale.setlocale(locale.LC_ALL, "de_DE")
print(my_date.strftime("%a %x")) # Mo 25.12.2023

# Calendar week
week = my_date.isocalendar()[1]
print("KW%02d" % week)

# first day of quarter
def get_first_day_of_the_quarter(my_date: dt.date) -> dt.date:
return dt.date(my_date.year, 1 + 3 * ((my_date.month - 1) // 3), 1)

====rounding datetimes to minutes====
def floor_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Floor (=round down) minutes to X min resolution."""
min_new = res * (my_dt.minute // res)
return my_dt.replace(minute=min_new, second=0, microsecond=0)

def ceil_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Ceil (=round up) minutes to X min resolution."""
min_new = res * (1 + my_dt.minute // res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

def round_dt_minutes(my_dt: dt.datetime, res: int = 5) -> dt.datetime:
"""Cound minutes to X min resolution."""
min_old_dec = float(my_dt.minute) + float(my_dt.second * 60)
min_new = res * round(min_old_dec / res)
return my_dt.replace(minute=0, second=0, microsecond=0) + dt.timedelta(
minutes=min_new
)

my_dt = dt.datetime.fromisoformat("2011-11-04 00:05:23+00:00")
print(f"original: {my_dt}")
print(f"floored: {floor_dt_minutes(my_dt,5)}")
print(f"ceileded: {ceil_dt_minutes(my_dt,5)}")
print(f"rounded: {round_dt_minutes(my_dt,5)}")

====Timing / Time elapsed====
import time
timestart = time.time()
sec = time.time() - timestart
print(f"Time elapsed: {sec:.2f} sec")

==OS, Argpars, etc.==
===Hostname===
from socket import gethostname
print(gethostname())

===Checking Operating System===
import os
import sys

if os.name == "posix":
print("posix/Unix/Linux")
elif os.name == "nt":
print("windows")
else:
print("unknown os")
sys.exit(1) # throws exception, use quit() to close / die silently

===Get filename of python script===
my_file_path = __file__

alternative using sys package
from sys import argv
myFilename = argv[0]

===accessing os envrionment variables===
import os
print(os.getenv("tmp"))
# better:
try:
s = os.environ[key] # throws error if unset
except KeyError:
error_message(f"Environment variable {key} not set.")

===Command Line Arguments===
====ArgumentParser====
import argparse

parser = (
argparse.ArgumentParser()
) # construct the argument parser and parse the arguments
# -h comes automatically

# Boolean Parameter
parser.add_argument(
"-v", "--verbose", help="increase output verbosity", action="store_true"
) # store_true -> Boolean Value

# Choice Parameter
# restrict to a list of possible values / choices
# parser.add_argument("--choice", type=int, choices= [0, 1, 2], help="Test choices")

# Positional Parameter (like text.py 123)
# parser.add_argument("num", type=int, help="Number of things")

# Required Parameter
# parser.add_argument("-i", "--input", type=str, required=True, help="Path of file")

# Optional Parameter
parser.add_argument("-n", "--number", type=int, help="Number of clicks")
# Optional Parameter with Default
parser.add_argument(
"-s",
"--seconds",
type=int,
default=sec_default,
help="Duration of clicking, default = %i (sec)" % sec_default,
)

args = vars(parser.parse_args())

if args["verbose"]:
pass # do nothing
# print ("verbosity turned on")
if args["number"]:
print("num=%i" % args["number"])

==== match case statement ====
(new in python 3.10)
match args:
case {"sap": "prod", "version": 1}:
...
case {"sap": "prod", "version": 2}:
...
case _: # default case

==File Access==
===Pathlib===
for migrating see table [https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module]
from pathlib import Path

p = Path("dir/test.txt")

log_file = Path(__file__).with_suffix(".log") # __file__ is name of python script

my_fname = p.name # alternative to basename
my_ext = p.suffix
(filepath_without_ext, file_ext) = (p.stem, p.suffix)
my_dir = p.parent
my_parent_dir = p.parents[1]
p2 = p.with_suffix(".json")
p3 = p.parent / (p.name + "-autofix.tex")

# checks
Path(my_file_str).exists()
Path(my_file_str).is_file()
Path(my_file_str).is_dir()

# loop over glob of files matching wildcard
for file_out in Path("mydir").glob("*-autofix.tex"):

# loop over dirs
p = Path() # cwd
list_of_repos = [x for x in p.iterdir() if x.is_dir() and (x / ".git").is_dir()]

alternative using old os functions
Split path into folder, filename, ext
import os
(dirName, fileName) = os.path.split(f)
(fileBaseName, fileExtension) = os.path.splitext(fileName)
fileOut = os.path.splitext(fileIn)[0] + "-out.txt"

===File Manipulations: copy, move, delete, touch===
from pathlib import Path

# copy
from shutil import copyfile
copyfile(Path("file-source.txt"), Path("dir") / Path("file-target.txt"))

# move / rename
Path("file.txt").replace(Path("file2.txt"))

# delete
Path("file.txt").unlink()

# touch
Path("file.txt").touch()

===File size and timestamp===
# size
Path("file.txt").stat().st_size

# timestamp last modified
Path("file.txt").stat().st_mtime

===Dir create/mkdir and delete===
# create dir
from pathlib import Path
Path("myDir1/myDir2").mkdir(parents=True, exist_ok=True)

# delete dir
import shutil
shutil.rmtree(d)

===Loop over Directories===
====Fetch Dir Contents / Loop over Files====
glob via pathlib
for fileOut in Path("mydir").glob("*-autofix.tex"):

Get list of files in directory, filter dirs from list, filter by ext
dir= "/path/to/some/dir"
listoffiles = [ f for f in os.listdir(dir) if os.path.isfile(os.path.join(dir ,f)) and f.lower()[-4:] == ".gpx"]
listoffiles.sort()

====Traverse in Subdirs====
# walk into path an fetch all files matching extension jpe?g
files = []
for (dirpath, dirnames, filenames) in os.walk("."):
dirpath = dirpath.replace("\\", "/")
for file in filenames:
if file.endswith(".txt"):
files.append(dirpath + "/" + file)
elif re.search(r"\.jpe?g$", file, re.IGNORECASE):
files.append(dirpath + "/" + file)

==File Parsing==
===File General===
====File Read====
path_file_in = Path("file.txt")

# via pathlib
cont = path_file_in.read_text(encoding="utf-8") # note: utf-8-sig for UTF-8-BOM

# general
with path_file_in.open(encoding="utf-8") as fh:
cont = fh.read()
# or
lst = fh.readlines()
# or
line = fh.readline()
# or
for line in fh:
print(line)

fh = path_file_in.open(encoding="utf-8")
...
fh.close()

====File Write====
path_file_out = Path("out/1/out.txt")

# write via pathlib
path_file_out.write_text("some text")

# write general
with path_file_out.open(mode="w", encoding="utf-8", newline="\n") as fh:
# w = overWrite file ; a = append to file
# If running Python in Windows, "\n" is automatically replaced by "\r\n".
# To prevent this use newline='\n'
fh.writelines(lst) # no linebreaks
# or
fh.write("\n".join(lst))
# or
for line in lst:
fh.write(line)

# Force update of file contents without closing it
fh.flush()

# alternative
fh = path_file_out.open(mode="w", encoding="utf-8", newline="\n")
...
fh.close()

===CSV===
====CSV Read====
import csv
from pathlib import Path

with Path("data.csv").open(mode="r", encoding="utf-8") as fh: # for Excel use ANSI
csv_reader = csv.DictReader(fh, dialect="excel", delimiter="\t")
for row in csv_reader:
print(f'\t{row["name"]} works in the {row["department"]} department')

====CSV Write====
import csv
from pathlib import Path

# simple
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.writer(fh, delimiter="\t")
csvwriter.writerow(("Date", "Confirmed"))

# dictwriter
with Path("data.tsv").open(mode="w", encoding="utf-8", newline="\n") as fh:
csvwriter = csv.DictWriter(
fh,
delimiter="\t",
extrasaction="ignore",
fieldnames=["date", "occupied_percent", "occupied", "total"],
)
csvwriter.writeheader()
for d in my_list_of_dicts:
d["occupied_percent"] = round(100 * d["occupied"] / d["total"], 1)
csvwriter.writerow(d)

===JSON===
====JSON Read====
import json
# from file
with path_to_download_file.open(encoding="utf-8") as fh:
d_json = json.load(fh)
# from string
d_json = json.loads(response_text)

def json_read(file_path: Path) -> list[dict[str, str]]:
"""
Read JSON data from file.
"""
with file_path.open(encoding="utf-8") as fh:
json_data = json.load(fh)
return json_data

====JSON Write====
Write dict to file in JSON format, keeping utf-8 encoding
import json

with path_to_download_file.open(mode="w", encoding="utf-8", newline="\n") as fh:
json.dump(my_dict, fh, ensure_ascii=False, sort_keys=False, indent=2)

def json_write(file_path: Path, json_data: list[dict[str, str]]) -> None:
"""
Write JSON data to file.
"""
with file_path.open("w", encoding="utf-8", newline="\n") as fh:
json.dump(json_data, fh, ensure_ascii=False, sort_keys=False, indent=2)

===Excel===
====Excel Read====
import openpyxl
# or xlsxwriter for write-only

workbook = openpyxl.load_workbook(
pathToMyExcelFile,
data_only=True, # read values instead of formulas
read_only=True, # suppresses: "UserWarning: wmf image format is not supported so the image is being dropped"
)
sheet = workbook["mySheetName"]
# or fetch active sheet
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=34, column=1) # index start here with 1
print(cell.value)
# or
print(sheet.cell(column=col, row=row).value)

====Excel Write====
import openpyxl

workbook = openpyxl.Workbook()
sheet = workbook.active
cell = sheet["A34"]
# or
cell = sheet.cell(row=i, column=j) # index starts at 1
cell.value = "asdf"
workbook.save("out.xlsx")

===Word===
====Word Read====
from io import BytesIO
from pathlib import Path
import pandas as pd
from docx import Document
from docx.document import Document as DocxDocument

def read_doc_to_df(p_doc: Path | BytesIO) -> pd.DataFrame: # noqa: C901, PLR0912
"""Read Word document to DataFrame."""
if isinstance(p_doc, Path):
doc = Document(str(p_doc))
elif isinstance(p_doc, BytesIO):
doc = Document(p_doc)

# list of multiple paragraphs per key
data: dict[str, dict[str, list[str]]] = {}

sections1: list[str] = []
section1 = ""
key = ""
section2 = ""
score = ""

for paragraph in doc.paragraphs:
assert paragraph.style is not None
text = paragraph.text.strip()
if paragraph.style.name == "Title": # doc title
continue
if paragraph.style.name == "Heading 1": # sheet name
section1 = text
if section1 not in sections1:
sections1.append(section1)
key = ""
section2 = ""
elif paragraph.style.name == "Heading 2": # Key of the question
key = text.split(" | ")[0]
key = section1 + "|" + key
section2 = ""
elif paragraph.style.name == "Normal" and text.startswith("[") and "]" in text:
section2, t2 = text[1:].split("]", maxsplit=1)
if section2 == "Score":
score = t2.strip()
data[key][section2] = [score]
elif paragraph.style.name == "Normal":
text = paragraph.text.strip()
# skip requirements leftovers and empty paragraphs
if (section2 == "" and text.startswith("...")) or text == "":
continue
lst = data.setdefault(key, {}).setdefault(section2, [])
lst.append(text)
data[key][section2] = lst
else:
msg = f"Unexpected style: {paragraph.style.name} in paragraph: {text}"
raise ValueError(msg)

# flatten the text from list to str
data2: dict[str, dict[str, str]] = {}
for key, sections in data.items():
for section2, lst in sections.items():
s = "\n".join(lst)
if key not in data2:
data2[key] = {}
data2[key][section2] = s.strip()

df1 = pd.DataFrame.from_dict(data2, orient="index")
df1.index.name = "Key"
df1 = df1.reset_index()
df1[ [ "Sheet", "Key" ] ] = df1["Key"].str.split("|", n=1, expand=True)

return df1

====Word Write====
from io import BytesIO
from pathlib import Path
import pandas as pd from docx import Document
from docx.document import Document as DocxDocument
from docx.shared import Cm

LAYOUT_SMALL_MARGINS = True
if LAYOUT_SMALL_MARGINS:
sections = doc.sections
for section in sections:
section.top_margin = Cm(1)
section.bottom_margin = Cm(1)
section.left_margin = Cm(1)
section.right_margin = Cm(1)

doc.add_heading("My Title", level=0)
doc.add_heading("My Section", level=1)
doc.add_paragraph("Some Text)
p = doc.add_paragraph()
runner = p.add_run(bold text")
runner.bold = True # runner.italic = True

==Regular Expressions==
See [http://docs.python.org/library/re.html]

See [https://pythex.org] for an online tester

multiple flags are joined via pipe |
s = re.sub("asdf.*", r"qwertz", s, flags=re.DOTALL | re.IGNORECASE)

====Lookahead and Lookbehind====
pos lookahead: (?=...)
neg lookahead: (?!...)
pos lookbehind (?<=...)
neg lookbehind (?<!...)

====matching====
import re

# V0: simple 1
myPattern = "(/\*\*\* 0097_210000_0192539580000_2898977_0050 \*\*\*/.*?)($|/\*\*\*)"
myRegExp = re.compile(myPattern, re.DOTALL)
myMatch = myRegExp.search(cont)
assert myMatch != None, f"golden file not found in file {filename}"
cont_golden = myMatch.group(1)

# V1: simple 2
assert (
re.match("^[a-z]{2}$", d_settings["country"]) != None
), f'Error: county must be 2 digit lower case. We got: {d_settings["country"]}'

# V2: find and count
was = r"""Part: ([^<+])"""
cnt_parts = len(re.findall(was, cont))
cont = re.sub(was, r"\n\nTeil: \1\n\n", cont)
assert cnt_parts == 4, f"{cnt_parts} == 4"
assert "Part:" not in cont

=====Match email=====
def checkValidEMail(email: str) -> bool:
# from https://stackoverflow.com/posts/719543/timeline bottom edit
if not re.fullmatch(r"^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$", email):
print("Error: invalid email")
quit()
return True

====Find all====
myMatches = re.findall('href="([^"]+)"', cont)
for myMatch in myMatches:
print(myMatch)

====substring====
import re

# simple via search
lk_id = re.search('^.*timeseries\-(\d+)\.json$', f).group(1)

# simple via sub
myPattern = "^.*" + s1 + "(.*)" + s2 + ".*$"
out = re.sub(myPattern, r"\1", s)

# more robust including an assert
def substr_between(s: str, s1: str, s2: str) -> str:
"""
returns substring of s, found between strings s1 and s2
s1 and s2 can be regular expressions
"""
myPattern = s1 + '(.*)' + s2
myRegExp = re.compile(myPattern)
myMatches = myRegExp.search(s)
assert myMatches != None, f"E: can't find '{s1}'...'{s2}' in '{s}'"
out = myMatches.group(1)
return out

matchObj = re.search(r"(\d+\.\d+)", text
if matchObj:
price = float( '%s' % (matchObj).group(0) )

====Naming of match groups====
(?P<name>...), see [https://docs.python.org/3/library/re.html]

====Search and Replace====
From [http://www.regular-expressions.info/python.html] 
re.sub(regex, replacement, str) performs a search-and-replace across subject, replacing all matches of regex in str with replacement. The result is returned by the sub() function. The str string you pass is not modified.

s = re.sub(" +", " ", s)

====Splitting====
From [http://docs.python.org/library/re.html] 
split() splits a string into a list delimited by the passed pattern. The method is invaluable for converting textual data into data structures that can be easily read and modified by Python as demonstrated in the following example that creates a phonebook.

First, here is the input. Normally it may come from a file, here we are using triple-quoted string syntax:
>>> input = """Ross McFluff: 834.345.1254 155 Elm Street
...
... Ronald Heathmore: 892.345.3428 436 Finley Avenue
... Frank Burger: 925.541.7625 662 South Dogwood Way
...
...
... Heather Albrecht: 548.326.4584 919 Park Place"""

The entries are separated by one or more newlines. Now we convert the string into a list with each nonempty line having its own entry:
>>> entries = re.split("\n+", input)
>>> entries
['Ross McFluff: 834.345.1254 155 Elm Street',
'Ronald Heathmore: 892.345.3428 436 Finley Avenue',
'Frank Burger: 925.541.7625 662 South Dogwood Way',
'Heather Albrecht: 548.326.4584 919 Park Place']

Finally, split each entry into a list with first name, last name, telephone number, and address. We use the maxsplit parameter of split() because the address has spaces, our splitting pattern, in it:
>>> [re.split(":? ", entry, 3) for entry in entries]
[['Ross', 'McFluff', '834.345.1254', '155 Elm Street'],
['Ronald', 'Heathmore', '892.345.3428', '436 Finley Avenue'],
['Frank', 'Burger', '925.541.7625', '662 South Dogwood Way'],
['Heather', 'Albrecht', '548.326.4584', '919 Park Place']]

==== replace cont by linebreaks====
def replace_cont_by_linebreaks(s: str, regex: str) -> str:
"""
Replace regex in s by the number of linebreaks it originally contained.
"""
myMatches = re.findall(regex, s, flags=re.DOTALL)
for match in myMatches:
linebreaks = match.count("\n")
s = s.replace(match, "\n" * linebreaks, 1)
return s

==Image/Picture/Photo==
from PIL import Image, ImageFilter # pip install Pillow

fileIn = "2018-02-09 13.56.25.jpg"
# Read image
img = Image.open(fileIn)
PROBLEM: 
PIL Image.save() drops the IPTC data like tags, keywords, copywrite, ... 
better using https://imagemagick.org instead when tags shall be kept

====Resize====
# Resize keeping aspect ration -> img.thumbnail
# drops exif data, exif can be added from source file via exif= in save, see below
size = 1920, 1920
img.thumbnail(size, Image.ANTIALIAS)

====Export file====
fileOut = os.path.splitext(fileIn)[0] + "-edit.jpg"
try:
img = Image.open(fileIn)
img.save(fp=fileOut, format="JPEG", quality='keep') # exif=dict_exif_bytes
# JPEG Parameters
# * qualitiy : 'keep' or 1 (worst) to 95 (best), default = 75. Values above 95 should be avoided.
# * dpi : tuple of integers representing the pixel density, (x,y)
except IOError:
print("cannot write file '%s'" % fileOut)

====Export Progressive / web optimized JPEG====
from PIL import ImageFile # for MAXBLOCK for progressive export
fileOut = os.path.splitext(fileIn)[0] + "-progressive.jpg"
try:
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)
except IOError:
ImageFile.MAXBLOCK = img.size[0] * img.size[1]
img.save(fp=fileOut, format="JPEG", quality=80, optimize=True, progressive=True)

====JPEG Meta Data: EXIF and IPTC====
=====IPTC: Tags/Keywords=====
from iptcinfo3 import IPTCInfo # this works in pyhton 3!
iptc = IPTCInfo(fileIn)
if len(iptc['keywords']) > 0: # or supplementalCategories or contacts
print('====> Keywords')
for key in sorted(iptc['keywords']):
s = key.decode('ascii') # decode binary strings
print(s)

=====EXIF via piexif=====
import piexif # pip install piexif
exif_dict = piexif.load(img.info['exif'])
print(exif_dict['GPS'][piexif.GPSIFD.GPSAltitude])
# returns list of 2 integers: value and donator -> v / d
# (340000, 1000) => 340m
# (51, 2) => 25.5m

# Modify altitude
exif_dict['GPS'][piexif.GPSIFD.GPSAltitude] = (140, 1) # 140m

# write to file
exif_bytes = piexif.dump(exif_dict)
fileOut = os.path.splitext(fileIn)[0] + "-modExif.jpg"
try:
img.save(fp=fileOut, format="jpeg", exif=exif_bytes, quality='keep')
except IOError:
print("cannot write file '%s'" % fileOut)
or
exif_dict = piexif.load(fileIn)
for ifd in ("0th", "Exif", "GPS", "1st"):
print("===" + ifd)
for tag in exif_dict[ifd]:
print(piexif.TAGS[ifd][tag]["name"], "\t",
tag, "\t", exif_dict[ifd][tag])
print(exif_dict['0th'][306]) # 306 = DateTime

=====EXIF via exifread=====
# Open image file for reading (binary mode)
fh = open(fileIn, "rb")
# Return Exif tags
exif = exifread.process_file(fh)
fh.close()
# for tag in exif.keys():
# if tag not in ('JPEGThumbnail', 'TIFFThumbnail', 'Filename', 'EXIF MakerNote'):
# print("%s\t%s" % (tag, exif[tag]))
print(exif["Image DateTime"])
print(exif["GPS GPSLatitude"])
print(exif["GPS GPSLongitude"])

=====EXIF GPS via PIL=====
# from https://developer.here.com/blog/getting-started-with-geocoding-exif-image-metadata-in-python3
def get_exif(filename):
image = Image.open(filename)
image.verify()
image.close()
return image._getexif()

def get_labeled_exif(exif):
labeled = {}
for (key, val) in exif.items():
labeled[TAGS.get(key)] = val
return labeled

def get_geotagging(exif):
if not exif:
raise ValueError("No EXIF metadata found")
geotagging = {}
for (idx, tag) in TAGS.items():
if tag == "GPSInfo":
if idx not in exif:
raise ValueError("No EXIF geotagging found")
for (key, val) in GPSTAGS.items():
if key in exif[idx]:
geotagging[val] = exif[idx][key]
return geotagging

def get_decimal_from_dms(dms, ref):
degrees = dms[0][0] / dms[0][1]
minutes = dms[1][0] / dms[1][1] / 60.0
seconds = dms[2][0] / dms[2][1] / 3600.0
if ref in ["S", "W"]:
degrees = -degrees
minutes = -minutes
seconds = -seconds
return round(degrees + minutes + seconds, 5)

def get_coordinates(geotags):
lat = get_decimal_from_dms(geotags["GPSLatitude"], geotags["GPSLatitudeRef"])
lon = get_decimal_from_dms(geotags["GPSLongitude"], geotags["GPSLongitudeRef"])
return (lat, lon)

exif = get_exif(fileIn)
exif_labeled = get_labeled_exif(exif)
print(exif_labeled["DateTime"])

geotags = get_geotagging(exif)
print(get_coordinates(geotags))

====Template Matching / Image Regocnition Using CV2 / OpenCV====
see [[Python - CV2]]

====Optical Character Recognition (OCR) ====
see [[Python - OCR]]

==Templates/Snippets==
===Retry on Exception===
for retry_no in range(MAX_RETRIES):
try:
if retry_no > 0:
logger.warning("retrying %d/%d...", retry_no + 1, MAX_RETRIES)
doit()
except json.JSONDecodeError:
logger.exception("JSONDecodeError")
if retry_no == MAX_RETRIES - 1:
# end of retries
raise

===Diff of 2 files===
import difflib

fh1 = p_file1.open("r", encoding="utf-8")
fh2 = p_file1.open("r", encoding="utf-8")
diff = difflib.ndiff(fh1.readlines(), fh2.readlines())
delta = "".join(line for line in diff if line.startswith(("+ ", "- ")))
print(delta)

===GPX parsing===
import gpxpy
import gpxpy.gpx
# Elevation data by NASA: see lib at https://github.com/tkrajina/srtm.py
fh_gpx_file = open(gpx_file_path, 'r')
gpx = gpxpy.parse(fh_gpx_file)
# Loops for accessing the data
for track in gpx.tracks:
for segment in track.segments:
for point in segment.points:
for waypoint in gpx.waypoints:
for route in gpx.routes:
for point in route.points:
# interesting properties of point / waypoint objects:
point.time
point.latitude
point.longitude
point.source
waypoint.name

===TypeHints===
from typing import Any, Dict, cast
creds = cast(dict[str, str], tomllib.load(f)) # type: ignore
o = cast(Dict[str, Any], tomllib.load(f)) # type: ignore
o["sap"] = cast(Dict[str, str], o["sap"])
o["settings"] = cast(Dict[str, str | int | bool], o["settings"])
o["settings"]["sleep_time"] = cast(int, o["settings"]["sleep_time"])

===Pylance/Pyright===
import tomllib
# shows warning: Import "xyz" could not be resolved
# fix by
import tomllib # pyright: ignore

===asserts function argument validation===
aus [https://bildungsportal.sachsen.de/opal/auth/RepositoryEntry/12518195218/CourseNode/94506982489539?0 Python Kurs von Carsten Knoll]
def eine_funktion(satz, ganzzahl, zahl2, liste):
if not type(satz) == str:
print "Datentpyfehler: satz"
return -1
if not isinstance(ganzzahl, int):
print "Datentpyfehler: ganzzahl"
return -2
if not isinstance(liste, (tuple, list)):
print "Datentpyfehler: liste"
return -3
# Kompakteste Variante (empfohlen):
assert zahl2 > 0, "Error: zahl2 ist nicht > 0" # Assertation-Error bei Nichterfuellung

def F(x):
if not isinstance(x, (float, int)):
msg = "Zahl erwartet, %s bekommen" % type(x)
raise ValueError(msg)
return x**2
better:
def F(x):
assert isinstance(x, (float, int)), "Error: x is not of type float or int"
return x**2

assert variant in [
"normal",
"gray",
"cannyedge",
], "Error: variant is not in 'normal', 'gray', 'cannyedge'"

===Exceptions===
see [https://medium.com/@saadjamilakhtar/5-best-practices-for-python-exception-handling-5e54b876a20]

====Catching====
Catch keyboard interrupt and do a "save exit"
try:
i = 0
while 1:
i += 1
print(i)
except KeyboardInterrupt:
print("stopped")

Catch '''all''' exceptions
try:
[...]
except Exception as e:
print("Exception: ", e)

====Raising Exceptions====
if resp.status_code != 200: # noqa: PLR2004
msg = f"Bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise ValueError(msg) from None

Custom Exceptions
try:
raise Exception("HiHo")

raise ValueError¶

===perl grep and map===
from [https://stackoverflow.com/questions/12845288/grep-on-elements-of-a-list]
def grep(list, pattern):
expr = re.compile(pattern)
return [elem for elem in list if expr.match(elem)]
or
filteredList = filter(lambda x: x < 7 and x > 2, unfilteredList)

def map(list, was, womit):
return list(map(lambda i: re.sub(was, womit, i), list))
# was = '.*"(\d+)".*'
# womit = r"\1"

===unit testing using pytest===
install via
pip install pytest
activate in vscode, see [https://code.visualstudio.com/docs/python/testing#_enable-a-test-framework]: To enable testing, use the Python: Configure Tests command on the Command Palette.

see https://docs.pytest.org/en/6.2.x/assert.html#assert

test_1.py:
import myLib # my custom lib to test

class TestClass:
def test_one(self):
x = "this"
assert "h" in x

def test_two(self):
assert myLib.multiply(1, 2) == 2

def test_three(self):
assert myLib.multiply(2, 2) == 2

====project structure====
=====simplest structore: test next to script=====
Alternative without tests dir:
./main.py
./helper.py
./helper_test.py
with helper_test.py:
from helper import fnc1

=====standalone model with src and tests dirs=====
for this dir structure
src/helper.py
tests/helper_test.py
helper_test.py needs
import sys
from pathlib import Path
sys.path.insert(0, (Path(__file__).parent.parent / "src").as_posix())

=====standalone model with tests dir=====
if the test files are placed inside a ./tests/ dir
./main.py
./helper.py
tests/helper_test.py
helper_test.py needs
sys.path.insert(0, Path(__file__).parent.parent.as_posix())
from helper import fnc1

====parametrize====
import pytest
@pytest.mark.parametrize(
("test_input", "expected"),
[("", None), ("PT30M", 30), ("PT3H", 3 * 60), ("PT2H30M", 2 * 60 + 30)],
)
def test_task_est_to_minutes(test_input: str, expected: int) -> None:
assert task_est_to_minutes(test_input) == expected

=====fixures: prepare and cleanup test_data=====
to automatically run before and after each testcase
@pytest.fixture(autouse=True)
def _setup_tests(): # noqa: ANN202
cache_prepare_lists()
cache_prepare_tasks()

yield

cache_cleanup_test_data()

====Test coverage report====
pip install pytest-cov
# console output
pytest --cov

# html report in coverage_report/index.html with details per file
pytest --cov --cov-report=html:coverage_report
to exclude a code block from coverage report, add
# pragma: no cover

=== Sleep / Wait for input ===
sleep for a while
import time
time.sleep(60)

wait for user input
input("press Enter to close")

===Suppress Warnings===
see [https://docs.python.org/3/library/warnings.html#temporarily-suppressing-warnings]
import warnings
warnings.filterwarnings("ignore", message=".*native_field_num.*not found in message.*")

===Logging===
====V4 minimal stdout logging====
import logging
logging.addLevelName(logging.DEBUG, "D")
logging.addLevelName(logging.INFO, "I")
logging.addLevelName(logging.WARNING, "W")
logging.addLevelName(logging.ERROR, "E")
logging.addLevelName(logging.CRITICAL, "C")
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(name)s: %(message)s")
LOGGER = logging.getLogger(__name__)
LOGGER.setLevel(logging.INFO)
logging.getLogger("httpx").setLevel(logging.WARNING)

# usage of variables and rounding of numbers. (%d also rounds floats)
LOGGER.info("file %s has %d lines with avg %.1f char/line", filename, lines, c_p_l)

====V3 simple logging====
# main file:
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(name)s\t%(message)s",
handlers=[
RotatingFileHandler(
Path(__file__).with_suffix(".log"),
maxBytes=10485760, # 10 MB = 10*1024*1024,
backupCount=1,
encoding="utf-8",
)
],
datefmt="%Y-%m-%d %H:%M:%S",
)
logger = logging.getLogger(Path(__file__).stem)
logger.info("Start")

# other files:
import logging
logger = logging.getLogger(__name__)

====V2: rotating file and STDOUT====
# 1. setup
import logging
from logging.handlers import RotatingFileHandler

logfile = "myApp.log"
maxBytes = 20 * 1024 * 1024
backupCount = 5
loglevel_console = logging.INFO
loglevel_file = logging.DEBUG

# create logger
logger = logging.getLogger("root")
logger.setLevel(loglevel_file)

# console handler
ch = logging.StreamHandler()
ch.setLevel(loglevel_console)

# rotating file handler
# fh = logging.FileHandler(logfile)
fh = RotatingFileHandler(logfile, maxBytes=maxBytes, backupCount=backupCount)
fh.setLevel(loglevel_file)

# create formatter and add it to the handlers
# %(name)s = LoggerName, %(threadName)s = TreadName
formatter = logging.Formatter(
"%(asctime)s - %(levelname)s - %(name)s - %(threadName)s - %(message)s "
)
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(ch)
logger.addHandler(fh)

logger.debug("DebugMe")
logger.info("Starting")
logger.warning("Attention")
logger.error("Something went wrong")
logger.critical("Something seriously went wrong ")

# 2. in other files/modules now use
import logging

logger = logging.getLogger(__name__)
logger.info("text")

...
except Exception as e:
logger.exception("Unhandeled exception")
quit()

====V1: Simple====
import logging
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s\t%(levelname)s\t%(message)s", # \t%(name)s
filename=Path(__file__).with_suffix(".log"), # uncomment to switch to stdout
filemode="a",
)
logger = logging.getLogger(__name__)
logger.debug("Some text")
logger.info("Some text")
logger.warning("Some text")
logger.error("Some text")
logger.critical("Some text")
try:
...
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise

===Process Bar===
see [https://tqdm.github.io tqdm]
from tqdm import tqdm
for i in tqdm(range(10000)):
....
or
with tqdm(total=total_iterations, desc="Processing") as progress_bar:
...
progress_bar.update(1)

===CGI Web development===
# Print necessary headers.
print("Content-Type: text/html")
print()

# errors and debugging info to browser
import cgitb
cgitb.enable()

Access URL or Form Parameters
# V2 from https://www.tutorialspoint.com/python/python_cgi_programming.htm
import cgi
form = cgi.FieldStorage()
username = form.getvalue('username')
print(username)

# V1
import sys
import urllib.parse
query = os.environ.get('QUERY_STRING')
query = urllib.parse.unquote(query, errors="surrogateescape")
d = dict(qc.split("=") for qc in query.split("&"))
print(d)

====CGI Backend Returning JSONs====
#!/usr/local/bin/python3.6
# -*- coding: utf-8 -*-

import cgi
import json

# Print necessary headers.
print("Content-type: application/json")
print()

def get_form_parameter(para: str) -> str:
"asserts that a given parameter is set and returns its value"
value = form.getvalue(para)
assert value, f"Error: parameter {para} missing"
assert value != "", f"Error: parameter {para} missing"
return value

response = {}
response['status'] = "ok"

try:
action = get_form_parameter("action")
response['action'] = action
if action == "myAction":
...

except Exception as e:
response['status'] = "error"
d = {"type": str(type(e)), "text": str(e)}
response["exception"] = d

finally:
print(json.dumps(response))

==Databases==
===PostgreSQL===
====Improved helper function====
"""Helper functions for DB access."""

import datetime as dt
import logging

import psycopg2
import psycopg2.extras
from creds_db import credentials as creds_db

# Configure logging
logger = logging.getLogger(__name__)

connection: psycopg2.extensions.connection = None # type: ignore
cursor_query: psycopg2.extensions.cursor = None # type: ignore
cursor_write: psycopg2.extensions.cursor = None # type: ignore

def connect() -> (
tuple[
psycopg2.extensions.connection,
psycopg2.extensions.cursor,
psycopg2.extensions.cursor,
]
):
"""Connect to the database."""
try:
connection = psycopg2.connect(**creds_db)
cursor_query = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)
cursor_write = connection.cursor()
logger.info(
"Connected to database %s on host %s",
creds_db["database"],
creds_db["host"],
)
except psycopg2.DatabaseError:
logger.exception("Database connection error: %s")
raise
else:
return connection, cursor_query, cursor_write

def disconnect() -> None:
"""Disconnect from the database."""
try:
if cursor_query:
cursor_query.close()
if cursor_write:
cursor_write.close()
if connection:
connection.close()
logger.info("Disconnected from the database")
except psycopg2.DatabaseError:
logger.exception("Error during disconnection: %s")

def exists(url: str, valid_hours: float) -> bool | None:
"""
Check if a record exists.

Returns
-------
None for missing record
True for valid record
False for expired record

"""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=valid_hours)
sql = """
SELECT delete_at > %s AS "valid"
FROM my_table
WHERE url = %s LIMIT 1;
"""
cursor_query.execute(sql, (delete_at, url))
res = cursor_query.fetchone()
return res[0] if res else None

def query1(url: str) -> dict | None:
"""Execute a query."""
sql = """
SELECT *
FROM my_table
WHERE url = %s LIMIT 1
"""
cursor_query.execute(sql, (url,))
return cursor_query.fetchone() # type: ignore

def insert(url: str, response: str, delete_in_hours: float) -> None:
"""Insert a record."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=delete_in_hours)
sql = """
INSERT INTO my_table (url, response, delete_at)
VALUES (%s, %s, %s);
"""
cursor_write.execute(sql, (url, response, delete_at))
connection.commit()

def delete(url: str) -> None:
"""Delete a record."""
sql = """
DELETE FROM my_table
WHERE url = %s;
"""
cursor_write.execute(sql, (url,))
connection.commit()

def delete_expired(in_hours: float) -> None:
"""Delete records that expire in less than in_hours from now."""
delete_at = dt.datetime.now(tz=dt.UTC) + dt.timedelta(hours=in_hours)

sql = """
DELETE FROM my_table
WHERE delete_at < %s;
"""
cursor_write.execute(sql, (delete_at,))
connection.commit()

connection, cursor_query, cursor_write = connect()

====Basics====
import psycopg2
import psycopg2.extras

credentials = {
"host": "localhost",
"port": 5432,
"database": "myDB",
"user": "myUser",
"password": "myPwd",
}
connection = psycopg2.connect(**credentials)
cursor = connection.cursor(cursor_factory=psycopg2.extras.DictCursor)

l_bind_vars = [["A1", "A2"], ["B1", "B2"]]

sql = """
SELECT * FROM myTable
WHERE 1=1
AND status NOT IN ('CLOSED')
AND ColA = %s
AND ColB = %s
ORDER BY created DESC
"""
cursor.execute(sql, l_bind_vars)
d_data = dict(cursor.fetchone())

====export result to csv file====
sql1 = "SELECT * FROM table"
sql2 = "COPY (" + sql1 + ") TO STDOUT WITH CSV HEADER DELIMITER '\t'"
with open("out.csv", "w") as file:
cursor.copy_expert(sql2, file)

====ReadConfigFile to connect to PostgreSQL====
from [http://www.postgresqltutorial.com/postgresql-python/connect/]
database.ini
[postgresql]
host=dbhost
port=5432
database=dbname
user=dbuser
password=dbpass

from configparser import ConfigParser

def config(filename="database.ini", section="postgresql"):
parser = ConfigParser()
parser.read(filename)
# get section, default to postgresql
db = {}
if parser.has_section(section):
params = parser.items(section)
for param in params:
db[param[0]] = param[1]
else:
raise Exception(
"Section {0} not found in the {1} file".format(section, filename)
)
return db

main.py
import psycopg2
from config import config

def connect():
"""Connect to the PostgreSQL database server"""
conn = None
try:
params = config() # read connection parameters
print("Connecting to the PostgreSQL database...")
conn = psycopg2.connect(**params)
cur = conn.cursor() # create a cursor
print("PostgreSQL database version:")
cur.execute("SELECT version()") # execute a statement
db_version = cur.fetchone()
print(db_version)
cur.close()
except (Exception, psycopg2.DatabaseError) as error:
print(error)
finally:
if conn is not None:
conn.close()
print("Database connection closed.")

if __name__ == "__main__":
connect()

===SQL Lite / SQLite===
see page [[SQLite]]

===InfluxDB===
see [[InfluxDB]]

==Internet Access==
===Send E-Mails===
see [[Python - eMail]]

===Download data using browser UA / REST===
import requests

headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0",
}
resp = requests.get(
url,
headers=headers,
timeout=3, # timeout in sec, requests should always have a timeout!
# timeout=(1,3), # 1s connect-timout, 3s read-timeout
)
if resp.status_code != 200: # noqa: PLR2004
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download only if cache is too old====
import time
from pathlib import Path
import requests

def fetch_url_or_cache(file_cache: Path, url) -> str:
"""Fetch URL and cache in a file."""
if check_cache_file_available_and_recent(
file_path=file_cache, max_age=3600, verbose=False
):
with file_cache.open(mode="r", encoding="utf-8") as fh:
s = fh.read()
else:
s = fetch(url=url)
with file_cache.open(mode="w", encoding="utf-8", newline="\n") as fh:
fh.writelines(s)
return s

def check_cache_file_available_and_recent(
file_path: Path,
max_age: int = 3500,
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = False
if file_path.exists() and (time.time() - file_path.stat().st_mtime < max_age):
cache_good = True
return cache_good

# or
def check_cache_file_available_and_recent_verbose(
file_path: Path, max_age: int = 3600, *, verbose: bool = False
) -> bool:
"""Check if cache file exists and is recent."""
cache_good = True
if not file_path.exists():
if verbose:
print(f"No Cache available: {file_path}")
cache_good = False
if cache_good and time.time() - (time.time() - file_path.stat().st_mtime < max_age):
if verbose:
print(f"Cache too old: {file_path}")
cache_good = False
return cache_good

def fetch(url) -> str:
"""Fetch URL."""
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0", # noqa: E501
}
resp = requests.get(url, headers=headers, timeout=5)
if resp.status_code == 200: # noqa: PLR2004
return resp.content.decode("ascii")
else: # noqa: RET505
msg = f"E: bad response. status code:{resp.status_code}, text:\n{resp.text}"
raise Exception(msg) # noqa: TRY002

====Download using urllib.request====
from urllib.request import urlopen

url = "https://pomber.github.io/covid19/timeseries.json"
with urlopen(url) as response: # noqa: S310
response_text = response.read()

===HTML parsing and extracting of elements===
V2: via BeautifulSoup
from bs4 import BeautifulSoup # pip install beautifulsoup4

soup = BeautifulSoup(cont, features="html.parser")
my_element = soup.find("div", {"class": "user-formatted-inner"})
my_body = my_element.prettify()
# my_body = my_element.encode()
# my_body = str(my_element)

V1: via lxml and xpath
from lxml import html
import requests

page = requests.get(url)
tree = html.fromstring(page.content)
tbody_trs = tree.xpath("//*/tbody/tr")
l_rows = []
for tr in tbody_trs:
l_columns = []
if len(tr) != 15:
continue
for td in tr:
l_columns.append(td.text_content())
l_rows.append(list(l_columns))

===HTML entities to unicode===
import html
cont = html.unescape(cont)

==GUI Interactions==
===Take Screenshot===
import pyautogui # (c:\Python\Scripts\)pip install pyautogui
# pyautogui does only support screenshots on monitor #1
...
screenshot = pyautogui.screenshot()
# screenshot = pyautogui.screenshot(region=(screenshotX,screenshotY, screenshotW, screenshotH))
screenshot = np.array(screenshot)
# Convert RGB to BGR
screenshot = screenshot[:, :, ::-1].copy()

===Mouse Actions===
def clickIt(x,y,key="") :
x0, y0 = pyautogui.position()
if key != "": # crtl, shift
pyautogui.keyDown(key)
pyautogui.moveTo(x, y, duration=0.2)
pyautogui.click(x=x , y=y, button='left', clicks=1, interval=0.1)
if key != "": # crtl, shift
pyautogui.keyUp(key)
pyautogui.moveTo(x0, y0)

===Web Automation===
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

# from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time
import glob

class StravaUserMapDL():
def __init__(self):
self.driver = webdriver.Firefox()

def login(self):
driver = self.driver
url = "https://www.somewebpage.com"
email = "myemail"
password = "mypassword"
driver.get(url)

title = driver.title
urlIs = driver.current_url
cont = driver.page_source # as string
FILE = open(filename,"w") # w = overWrite file ; a = append to file
FILE.write(cont)
FILE.close()

# handle login if urlIs != url
if (urlIs != url):
# activate checkbox 'remember_me'
elem = driver.find_element_by_id('remember_me')
if (elem.is_selected() == False):
elem.click()
assert elem.is_selected() == True
elem = driver.find_element_by_id('email')
elem.send_keys(email)
elem = driver.find_element_by_id('password')
elem.send_keys(password)
elem.send_keys(Keys.RETURN)
# Wait until login pages is replaced by real page
urlIs = driver.current_url
while (urlIs != url):
time.sleep(1)
urlIs = driver.current_url
print (urlIs)

# results = driver.find_elements_by_class_name('following')
# results = driver.find_elements_by_tag_name('li')

# print(results[0].text)
assert (urlIs == url)

===Unit Tests using Web Automation===
import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

#from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options

import os
import time

class PythonOrgSearch(unittest.TestCase):
# def __init__(self,asdf):
# self.driver = webdriver.Firefox()

def setUp(self):
print ("setUp")
# headless mode:
# opts = Options()
# opts.set_headless()
# assert opts.headless # Operating in headless mode
# self.driver = webdriver.Firefox(options=opts)

self.driver = webdriver.Firefox()

def test_search_in_python_org(self):
driver = self.driver
driver.get("http://www.python.org")
self.assertIn("Python", driver.title)
elem = driver.find_element_by_name("q")
elem.send_keys("pycon")
elem.send_keys(Keys.RETURN)
assert "No results found." not in driver.page_source
print ("fertig: python_org")

def tearDown(self):
print ("tearDown")
print ("close Firefox")
self.driver.close() # close tab
self.driver.quit() # quit browser
# os._exit(1) # exit unittest without Exception

if __name__ == "__main__":
try:
unittest.main()
except SystemExit as e:
os._exit(1)

==Cryptography and Hashing==
====Hashing via SHA256====
def gen_SHA256_string(s: str) -> str:
m = hashlib.sha256()
m.update(s.encode("ascii"))
return m.hexdigest()

====Hashing via MD5====
(MD5 is not secure, better use SHA256)
def gen_MD5_string(s: str) -> str:
m = hashlib.md5()
m.update(s.encode("ascii"))
return m.hexdigest()

====Password hashing via bcrypt====
import bcrypt
pwd = 'geheim'
pwd = pwd.encode("utf-8")
# or
pwd = b'geheim'

hashed = bcrypt.hashpw(pwd, bcrypt.gensalt())
if bcrypt.checkpw(pwd, hashed):
print("It Matches!")
print(hashed.decode("utf-8"))

To use version 2a instead of 2b (default):
bcrypt.gensalt(prefix=b"2a")

==Multiprocessing, subprocesses and Threading==
see [[Python_-_Multithreading]] as well

use processes for CPU limited work 
use threads for I/O limited work

===Simple single process===
import subprocess
process = subprocess.run(["sudo", "du", "--max-depth=1", mydir], capture_output=True, text=True)
print (process.stdout)
old, depricated way:
os.system( "gnuplot " + gpfile)

===Multiprocessing===
see [[Python - Multithreading]] as well

V2 using pool and starmap
import multiprocessing
import os

def worker(i: int, s: str) -> list:
result = (i, s, os.getpid())
return result

if __name__ == "__main__":
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n" + str(i))
l_pile_of_work.append((tup))
# gen pool of processes
num_processes = min(multiprocessing.cpu_count(), len(l_pile_of_work))
pool = multiprocessing.Pool(processes=num_processes)
# start processes on pile of work
l_results_unsorted = pool.starmap(
func=worker, iterable=l_pile_of_work # each item is a list of 2 parameters
)

# or if only one parameter:
# l_results_unsorted = pool.map(doit_de_district, l_pile_of_work)

l_results = sorted(l_results_unsorted) # sort by i

V1
import subprocess
l_subprocesses = [] # queue list of subprocesses
max_processes = 4

def process_enqueue(new_process_parameters):
global l_subprocesses
# wait for free slot
while len(l_subprocesses) >= max_processes:
process_remove_finished_from_queue()
time.sleep(0.1) # sleep 0.1s
process = subprocess.Popen(new_process_parameters,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
universal_newlines=True)
l_subprocesses.append(process)

def process_remove_finished_from_queue():
global l_subprocesses
i = 0
while i <= len(l_subprocesses) - 1:
process = l_subprocesses[i]
if process.poll != None: # has already finished
process_print_output(process)
l_subprocesses.pop(i)
else: # still running
i += 1

def process_print_output(process):
"""waits for process to finish and prints process output"""
stdout, stderr = process.communicate()
if stdout != '':
print(f'Out: {stdout}')
if stderr != '':
print(f'ERROR: {stderr}')

def process_wait_for_all_finished():
global l_subprocesses
for process in l_subprocesses:
process_print_output(process)
l_subprocesses = [] # empty list of done subprocesses

process_enqueue(l_parameters1)
...
process_enqueue(l_parameters999)
process_wait_for_all_finished()

===Threading===
import threading
import queue
import os
import time

def worker(q_work: queue.Queue, results: dict):
while not q_work.empty():
i, s = q_work.get()
time.sleep(.1)
result = (i, s, os.getpid())
results[i] = result
q_work.task_done()

if __name__ == '__main__':
d_results = {} # threads can write into dict
# gen. pile of work
l_pile_of_work = []
for i in range(1_000):
tup = (i, "n"+str(i))
l_pile_of_work.append((tup))
# convert list of work to queue
q_pile_of_work = queue.Queue(
maxsize=len(l_pile_of_work)) # maxsize=0 -> unlimited
for params in l_pile_of_work:
q_pile_of_work.put(params)
# gen threads
num_threads = 100
l_threads = [] # List of threads, not used here
for i in range(num_threads):
t = threading.Thread(name='myThread-'+str(i),
target=worker,
args=(q_pile_of_work, d_results),
daemon=True)
l_threads.append(t)
t.start()
q_pile_of_work.join() # wait for all threas to complete
l_results_unsorted = d_results.values()
l_results = sorted(l_results_unsorted) # sort by i

===asyncio — Asynchronous I/O===
see https://docs.python.org/3/library/asyncio-task.html
import asyncio
import time

# basics
# task = asyncio.create_task(coro())
# Wrap the coro coroutine into a Task and schedule its execution. Return the Task object.

# sleep
# await asyncio.sleep(1)

# Running Tasks Concurrently and gathers the return values in list L
# L = await asyncio.gather(coro(x1,y1), coro(x2,y2), coro(x3,y3))

async def say_after(delay, what):
# Coroutines declared with the async/await syntax
await asyncio.sleep(delay)
print(what)

async def main():
# The asyncio.create_task() function to run coroutines concurrently as asyncio Tasks.
task1 = asyncio.create_task(
say_after(1, 'hello'))

task2 = asyncio.create_task(
say_after(1, 'world'))

print(f"started at {time.strftime('%X')}")

# Wait until both tasks are completed
await task1
await task2

print(f"finished at {time.strftime('%X')}")

asyncio.run(main())

==Pandas==
see [[Pandas]]

==Matplotlib==
see [[Matplotlib]]

== GUI via tkinter==
import tkinter as tk # no need to install via pip

class App(tk.Tk):
def __init__(self):
super().__init__()
self.title("robot-CClicker")
self.geometry("200x200+0+1080")
self.resizable(width=False, height=False)

self.l_buttons = []
self.__create_widgets()

def __create_widgets(self):
self.btn_click500 = tk.Button(
master=self,
text="500 clicks",
width=20,
command=lambda: self.clickBigCookie(500),
)
self.l_buttons.append(self.btn_click500)

for button in self.l_buttons:
button.pack(anchor=tk.W)

def clickBigCookie(self, num):
# TODO: the disabling of the buttons is not working
for button in self.l_buttons:
button["state"] = tk.DISABLED
helper.clickIt(self.posBigCockie[0], self.posBigCockie[1], num=num)
for button in self.l_buttons:
button["state"] = tk.NORMAL

if __name__ == "__main__":
app = App()
app.mainloop()

==Protobuf==
===convert .proto file to python class===
see https://www.datascienceblog.net/post/programming/essential-protobuf-guide-python/
protoc my_message.proto --python_out ./

===read/decode protobuf message===
parse message from file
import machine_message_pb2

with open("out.bin", "rb") as f:
my_message = my_message_pb2.my_message()
my_message.ParseFromString(f.read())
print(machine_message)

===create/encode protobuf message===
import machine_message_pb2

my_message = my_message_pb2.my_message()
my_message.data.field1 = 1
my_message.data.field2 = "asdf"

with open("out.bin", "wb") as f:
f.write(my_message.data.SerializeToString())

==venv / virtual environments==
pip install --upgrade pip
python -m venv .venv --prompt $(basename $(pwd))
source .venv/bin/activate
pip install -r requirements.txt
...
deactivate

==Pydantic data validation==
===Function input parameter validation===
see https://docs.pydantic.dev/usage/validation_decorator/
@validate_arguments
def sum(x: int, y: float) -> float:
print(x, y)
return x + y

print(sum(1.1, 1.1))
# -> 2.1

==Read config file==
===TOML===
see https://learnxinyminutes.com/docs/toml/

minimal example:
config.toml
# SAP API endpoint
[general]
sleep_time = 60
use_proxy = true
tool.py
try:
import tomllib # comes with python3.11
except ModuleNotFoundError:
import tomli as tomllib # pip install tomli
with open("config.toml", "rb") as f:
o = tomllib.load(f) # type: ignore
if o["settings"]["use_proxy"]:
...
for validation, see https://realpython.com/python-toml/

====TOML Write====
In for deployments I sometime want to modify a TOML config file for production, based on the dev version.
import tomli_w # pip install tomli-w
p_in = Path(__file__).parent.parent / ".streamlit/config.toml"
p_out = p_in.parent / "config-prod.toml"
with p_in.open("rb") as fh:
o = tomllib.load(fh)
o["server"]["fileWatcherType"] = "none"
del o["server"]["address"]
with p_out.open("wb") as fh:
tomli_w.dump(o, fh)

===ConfigParser: INI Config File Reading===
better use TOML

Config.ini
[Section1]
Cursor = 205E18
Grandma = 18E18
Farm = 11E18
Mine = 514E18
Factory = 155E18
test.py
from configparser import ConfigParser

config = ConfigParser(
interpolation=None
) # interpolation=None -> treats % in values as char % instead of interpreting it
config.read("Config.ini", encoding="utf-8")

print(config.getint("Section1", "key1"))
print(config.getfloat("Section1", "key2"))
print(config.get("Section1", "key3"))

for sec in config.sections():
d_settings = {}
for key in config.options(sec):
value = config.get(sec, key)
d_settings[key] = value
print("%15s : %s" % (key, value))

==VCard / vcf==
import codecs
import vobject # pip install vobject

obj = vobject.readComponents(codecs.open(file_in, encoding="utf-8").read()) # type: ignore
contacts: list[vobject.base.Component] = list(obj) # type: ignore

card = contacts[0]

if "bday" not in card.contents:
continue

# bday: remove 'VALUE': ['DATE']
card.contents["bday"][0].params = {} # type: ignore

# remove all fields but "bday", "n"
for key in card.contents.copy(): # loop over copy, to allow for deleting keys
if key not in ("n", "bday"):
del card.contents[key]

# recreate fn based on n
card.add("fn")
n = card.contents["n"][0]
fn = f"{n.value.given} {n.value.additional} {n.value.family}" # type: ignore
fn = re.sub(r"\s+", " ", fn)
card.fn.value = fn # type: ignore

with open("out.vcf", mode="w", encoding="utf-8", newline="\n") as fhOut:
fhOut.write(card.serialize())

==Sentry Exception monitoring==
see [[Sentry]]
==MQTT==
#!/usr/bin/env python3.12
"""Read power data from Tasmota device via MQTT."""

import json
from typing import Any

import paho.mqtt.client as mqtt
from mqtt_credentials import hostname, password, port, username
from paho.mqtt.reasoncodes import ReasonCode

def on_connect(
mqtt_client: mqtt.Client,
userdata: Any,
flags: dict[str, int],
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client connects MQTT broker.""" # noqa: D401
if reason_code == 0:
print("Connected to MQTT")
# print(f"MQTT protocol version: {mqtt_client._protocol}")

# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
mqtt_client.message_callback_add("tele/tasmota_MT681/SENSOR", on_message)
mqtt_client.subscribe([("tele/tasmota_MT681/SENSOR", 0)])

else:
print(f"Connection to MQTT broker failed. Error: {reason_code}")

def on_disconnect(
mqtt_client: mqtt.Client,
userdata: Any,
reason_code: ReasonCode,
properties: Any,
) -> None:
"""Callback when the client is disconnected from the MQTT broker.""" # noqa: D401
if reason_code > 0:
print(f"Unexpected disconnection. Error: {reason_code}")

def on_message(
mqtt_client: mqtt.Client,
userdata: Any,
message: mqtt.MQTTMessage,
) -> None:
"""Callback when a Tasmota message is received from the MQTT broker.""" # noqa: D401
s = message.payload.decode()
data = json.loads(s)
# {'Time': '2024-03-16T06:44:08', 'MT681': {'Total_in': 16.4396, 'Power_cur': 80, 'Total_out': 14.6403}} # noqa: E501

total_i = data["MT681"]["Total_in"]
total_o = data["MT681"]["Total_out"]

print(f"In:\t{total_i}\nOut:\t{total_o}")

mqtt_client.disconnect()
mqtt_client.loop_stop()

try:
# Create an MQTT client instance
mqtt_client: mqtt.Client = mqtt.Client(
mqtt.CallbackAPIVersion.VERSION2, "Python-Client-1"
) # type: ignore
mqtt_client.username_pw_set(username, password)

mqtt_client.on_connect = on_connect

# Connect to the MQTT broker
mqtt_client.connect(hostname, port, keepalive=60)

# NOT: while True, as that eats the CPU !!!
mqtt_client.loop_forever()
except KeyboardInterrupt:
print("KeyboardInterrupt, disconnecting from MQTT broker.")
mqtt_client.disconnect()
mqtt_client.loop_stop()

==Caching Functions==
from functools import lru_cache

@lru_cache(maxsize=128) # None equals to @cache declarator
def fnc(n:int):

==Performance/Resources==
===RAM/Memory consumption/bottleneck===
import tracemalloc
tracemalloc.start()

# ...
# Code
# example:
lst = list(range(1_000_000))
# ...

max_bytes = tracemalloc.get_traced_memory()[0]
print(f"{round(max_bytes / 10**6)}MB")

tracemalloc.stop()

===Profiling / Count and Runtime per Function===
call_stats = {}

def track_function_usage(func: Callable) -> Callable:
"""Annotation for gathering runtime statistics."""

@wraps(func)
def wrapper(*args: tuple, **kwargs: dict): # noqa: ANN202
start_time = time.time()
result = func(*args, **kwargs) # Call the original function
end_time = time.time()
call_stats[func.__name__]["calls"] += 1
call_stats[func.__name__]["total_time"] += end_time - start_time
return result
return wrapper

==JWT Token==
try:
decoded_token = jwt.decode(token, options={"verify_signature": False})
except jwt.ExpiredSignatureError:
msg = "Token has expired."
logger.exception(msg)
except jwt.InvalidTokenError:
msg = "Invalid token."
logger.exception(msg)

==Streamlit==
see https://github.com/entorb/streamlit-examples for a collection of my examples

===Minimum Example===
# src/app.py
from pathlib import Path
import streamlit as st
from streamlit.logger import get_logger
logger = get_logger(Path(__file__).stem)
logger.info("Start")
st.set_page_config(page_title="AppTitle", page_icon=None, layout="wide")
st.title("MyPageTitle")
st.header("MyHeader")
st.subheader("MySubHeader")
sel_param = st.selectbox("Parameter", ("count", "sum"))

# .streamlit/config.toml
[browser]
gatherUsageStats = false

[server]
port = 8501

# run it
streamlit run src/app.py

====Reduce Logging for K8s Deployed Docker Containers====
# reduce log output
for logger_name in [
"streamlit.runtime.caching.cache_utils",
"streamlit.runtime.caching.storage.in_memory_cache_storage_wrapper",
"streamlit.runtime.runtime",
"urllib3.connectionpool",
]:
get_logger(logger_name).setLevel(logging.WARNING)

===Chart via Altair===
====Line Chart====
chart = st.line_chart(df["value"])
# corresponds to
import altair as alt
c = (
alt.Chart(df)
.mark_line()
.encode(
x=alt.X("created", title=None),
y=alt.Y("value", title=None),
)
)
st.altair_chart(c, use_container_width=True)

====Bar Chart====
chart = (
alt.Chart(df)
.mark_bar()
.encode(
x=alt.X("yearmonth(date):T", title="Month"),
y=alt.Y(f"count:Q", title=None),
color="status:N",
tooltip=["yearmonth(date):T", "status:N", "cnt:Q"],
)
# .properties(width=600, height=400)
)
st.altair_chart(chart, use_container_width=True)

:T stands for Temporal. It indicates that the field contains date or time values.
:N stands for Nominal. It indicates that the field contains categorical data, which represents discrete categories or labels.
:Q stands for Quantitative. It indicates that the field contains numerical data that can be measured and compared.

===Excel Download===
def excel_download_buttons(df: pd.DataFrame, file_name: str = "export.xlsx") -> None:
"""Show prepare data and download buttons."""
if st.button(label="Click to prepare download"):
buffer = io.BytesIO()
with pd.ExcelWriter(buffer, engine="xlsxwriter") as writer:
df.to_excel(writer, sheet_name="Sheet1", index=False)
writer.close()

st.download_button(
label="Download data as Excel",
data=buffer,
file_name=file_name,
mime="application/vnd.ms-excel",
)

==UV Package Manager==
# create/update lockfile
uv lock --upgrade
# sync venv
uv sync --upgrade
# extract package info to requirements.txt
uv export --format requirements.txt --no-dev --no-hashes -o requirements.txt

Update Python version for my current project
uv venv --python=3.13.7

===Update UV===
uv self update
# or
brew update && brew upgrade uv

===Update Python Versions===
uv python upgrade

===UV + Docker on readOnlyRootFilesystem===
Example 1: https://github.com/SWE-Group-23/CM22007---Source/blob/10d02004f3a4208f7b30d37a0a87e89532a23575/create-service.sh#L117

Dockerfile
WORKDIR /app
COPY . .

ENV CASS_DRIVER_NO_EXTENSIONS=1

RUN --mount=type=cache,target=/root/.cache/uv \\
--mount=type=bind,source=uv.lock,target=uv.lock \\
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \\
uv sync --frozen --no-install-project

RUN uv pip install -e shared/

RUN addgroup -S group1 && adduser -S user1 -G group1 -u 1000
RUN chown -R user1:group1 /app
USER user1:group1

CMD ["uv", "run", "main.py"]

Deployment
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
runAsNonRoot: true
runAsUser: 1000
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault
volumeMounts:
- mountPath: /home/user1/.cache/uv
name: uv
- mountPath: /tmp
name: temp

Example 2: [https://hynek.me/articles/docker-uv/ Production-ready Python Docker Containers with uv]

Dockerfile (without the comments)
ENV UV_LINK_MODE=copy \
UV_COMPILE_BYTECODE=1 \
UV_PYTHON_DOWNLOADS=never \
UV_PYTHON=python3.12 \
UV_PROJECT_ENVIRONMENT=/app

RUN --mount=type=cache,target=/root/.cache \
--mount=type=bind,source=uv.lock,target=uv.lock \
--mount=type=bind,source=pyproject.toml,target=pyproject.toml \
uv sync \
--locked \
--no-dev \
--no-install-project

COPY . /src
WORKDIR /src
RUN --mount=type=cache,target=/root/.cache \
uv sync \
--locked \
--no-dev \
--no-editable

=== Vulture: Search for dead code ===
see [https://github.com/jendrikseipp/vulture GitHub]

pyproject.toml
[tool.vulture]
paths = ["src"]
# exclude = ["src/models.py"]
ignore_names = [
"LOGGER",
]

.pre-commit-config.yaml
# Vulture: find dead code
- repo: https://github.com/jendrikseipp/vulture
rev: v2.16
hooks:
- id: vulture

Uberspace

2026-04-08T05:44:30Z

Torben: /* Streamlit */

[[Category:Webserver]][[Category:Linux]]
==Manuals==
https://manual.uberspace.de and https://lab.uberspace.de

==PHP==
see https://manual.uberspace.de/lang-php/

to restart
uberspace tools restart php

===php-fpm===
less /opt/uberspace/etc/$USER/php-fpm.conf

==Python FastAPI==
see https://lab.uberspace.de/guide_fastapi/

# var 1: venv
python3.11 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn
pip install fastapi.responses
pip install gunicorn uvloop httptools
deactivate

# var 2: not use venv
pip3.11 install --user fastapi uvicorn fastapi.responses gunicorn uvloop httptools

# config backend
uberspace web backend set /strava-be --http --port 9001
# --remove-prefix
uberspace web backend list

supervisorctl restart fastapi

~/fastapi/conf.py
#!/usr/bin/env python3.10

# apply changes via
# supervisorctl restart fastapi

import os

app_path = os.environ["HOME"] + "/fastapi"

# Gunicorn configuration
wsgi_app = "main:api"
bind = ":9001"
chdir = app_path
workers = 1
worker_class = "uvicorn.workers.UvicornWorker"

# errorlog = app_path + "/errors.log"
# accesslog = app_path + "/access.log"

~/fastapi/main.py
#!/usr/bin/env python3.10

# apply changes via
# supervisorctl restart fastapi

from fastapi import FastAPI
from fastapi.responses import JSONResponse
from pydantic import BaseModel

import subprocess

class StravaSession(BaseModel):
sessionId: str

api = FastAPI()

@api.post("/strava-be/activityStats2/")
async def activityStats2(session: StravaSession):
# return {session}
response = {}
# response = {"session": session.sessionId}
process = subprocess.run(
[
"python3.10",
"/var/www/virtual/entorb/html/strava/activityStats2.py",
session.sessionId,
],
capture_output=True,
)

if process.returncode == 0:
response["status"] = "ok"
response_code = 200
# response["process_stdout"] = process.stdout.decode()
else:
response["status"] = "error"
response_code = 400
# response["process_stdout"] = process.stdout.decode()
response["error_message"] = process.stderr.decode()
return JSONResponse(content=response, status_code=response_code)

# To test this app locally, uncomment:
# import uvicorn
# uvicorn.run(api, host="localhost", port=8001)

# curl -i -X POST "https://entorb.net/strava-be/activityStats2/" -H "Content-Type: application/json" -d '{"sessionId": "1234"}'

==Checkliste neuer U7 Account==
~/.bashrc und ~/.bash_profile von altem angeglichen
Logs aktiviert
uberspace web log access enable
uberspace web log apache_error enable
uberspace web log php_error enable
uberspace web errorpage 500 disable

==Probleme bei der Migration Uberspace U6 zu U7==
===0. E-Mails===
Es scheinen keine E-Mails mehr rein und raus zu gehen. (Account xxx@entorb.net).

Neuer Webmailer ist https://webmail.uberspace.de , dieser zeigt aber erst nach Abschluss der Migration auf das neue Postfach. Solange die Migration nicht abgeschlossen ist, muss im Mail Client (wie Thunderbird) dieser Login für den U7 Mail Account verwendet werden:
statt xxx@entorb.net -> xxx@entorb.uber.space

===1. HTTPS Zertifikat===
HTTPS Zertifikat war auf einmal nicht mehr gültig für https://www.entorb.net . Das hatte erst auch auf dem U7 funktioniert, nun plötzlich nicht mehr. UPDATE: Problem verschwand von alleine, hing vielleicht mit dem DNS Umzug zusammen.

===2. bestehende ezmlm Mailinglisten===
ezmlm ist gemäß [https://lab.uberspace.de/guide_ezmlm.html Anleitung] installiert, allerdings sagt mir
ezmlm-make -+ ~/ezmlm/oldlist ~/.qmail-oldlist oldlist entorb.uber.space
> ezmlm-make: fatal: unable to stat /etc/ezmlm/de: file does not exist

Wenn ich hingegen testweise einen neuen Verteiler anlege tut es:
ezmlm-make -A -u -m -5 xxx@entorb.net ~/ezmlm/mylist ~/.qmail-mylist
mylist entorb.uber.space
und diesen kann ich dann auch bearbeiten via
ezmlm-make -+ ~/ezmlm/mylist ~/.qmail-mylist mylist entorb.uber.space

Lösung:
in ezmlm/oldlist/ezmlmrc die entsprechende Zeile löschen

===3. Zugriff auf Dateien außerhab von ~/html===
Ich hatte einige Dateien (wie zB eine sqlite DB) in meinem Home (~/) liegen. Auf diese sollen Skripte die sich unter
~/html->/var/www/virtual/entorb/html
befinden zugreifen können. Das funktionierte nicht mehr, da SELinux hier rein grätscht

Beste Lösung: Die Dateien auf die der httpd zugreifen soll nach
/var/www/virtual/entorb/
verschieben und im Home Symlink hinterlassen.

Erster Ansatz: der SELinux Kontexte lässt sich so setzen (Siehe [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-security-enhanced_linux-working_with_selinux-selinux_contexts_labeling_files]
).
chcon -t httpd_sys_content_t file-name
half hier aber nicht:
ls -laZ /home/ |grep ento
drwx------. entorb entorb unconfined_u:object_r:user_home_dir_t:s0 entorb

===4. Perl Module, wie zB Excel::Writer::XLSX===
Installiert via
cpanm Excel::Writer::XLSX

Führe ich das Skript ~/html/test.pl in der Shell aus, wird das Modul
gefunden und geladen. Im Browser via https://entorb.net/test.pl wird
das Modul nicht gefunden.

Ursache 4.1 war dass SELinux den Zugriff des httpd auf /home/entorb/perl5 blockierte

Lösung:
~/perl5 Verzeichnis nach
/var/www/virtual/entorb/perl5
verschoben und im Home einen Symlink hinterlassen.

Dann zuerst lib::local installiert, dann andere Module.

Ursache 4.2 war dass ich in der .bashrc dies gesetzt habe:
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
das musste ich im Perl Skript ebenfalls hinterlegen:
use lib ('/var/www/virtual/entorb/perl5/lib/perl5');
# aber !kein! use local::lib; !!!

===5. sendmail===
ich stellte fest, dass ich im U7 aus Python oder Perl CGI Skripten nicht mehr auf sendmail zugreifen kann. Über die Shell kommt die Mail sofort an. Beispiel:
8<---
import os
SENDMAIL = "/usr/sbin/sendmail"
to = "xxx@entorb.net"
subject = "testmail"
sender = "U7 <no-reply@entorb.net>"
body = "leer"
mail = f"To: {to}\nSubject: {subject}\nFrom: {sender}\nContent-Type:
text/plain; charset=\"utf-8\"\n\n{body}"
p = os.popen(f"{SENDMAIL} -t -i", "w")
p.write(mail)
p.close()
8<---
Lösung: statt Versand aus Web-Skripten/Seiten via sendmail, besser aus den Skripten ein Insert in eine SQLite DB machen und diese via cronjob und php mail() Funktion periodisch abarbeiten.

===6. Wiederholung der Migration===
Entweder einfach Migration auf dem U6 nochmal starten via
uberspace-move-account -u entorb
oder in Dashboard den Umzug abbrechen und neu starten -> anderer Server.

===7. MySQL DB Passwort angeblich zu kurz===
Das Migrationsskript hatte behauptet, dass mein MySQL Passwort zu kurz sei. Stimmte nicht, vermutlich kam es mit den Sonderzeichen durcheinander. Habe daher ein neues langes ohne Sonderzeichen vergeben.

==Streamlit==
see also [https://github.com/entorb/strava-streamlit/blob/main/README.md strava-streamlit readme]

Installation
pip3.11 install --user streamlit

Setup dir and config
mkdir ~/strava-streamlit
cd ~/strava-streamlit
mkdir .streamlit
vim .streamlit/config.toml

[browser]
gatherUsageStats = false

[server]
headless = true
port = 8501
baseUrlPath = "/strava-streamlit"

baseUrlPath is important when not running in web server document root

create minimal Streamlit app
vim src/app.py

import streamlit as st
st.title("Test")

start it manually as testrun (stop by ctrl+c)
streamlit run src/app.py

Add web backend
uberspace web backend set /strava-streamlit --http --port 8501
maybe add --remove-prefix

Access it via browser
https://entorb.net/strava-streamlit

Create Service
vim ~/etc/services.d/strava-streamlit.ini

[program:strava-streamlit]
directory=%(ENV_HOME)s/strava-streamlit
command=python3.11 -O -m streamlit run src/app.py
# for env var:
environment=MY_VAR="asdf"
loglevel=info

Start Service
supervisorctl reread
supervisorctl update
supervisorctl status
supervisorctl restart strava-streamlit

Check log
supervisorctl tail -f strava-streamlit
# as this is empty:
tail -f ~/logs/supervisord.log

Uberspace

2026-04-08T05:44:01Z

Torben: /* Streamlit */

[[Category:Webserver]][[Category:Linux]]
==Manuals==
https://manual.uberspace.de and https://lab.uberspace.de

==PHP==
see https://manual.uberspace.de/lang-php/

to restart
uberspace tools restart php

===php-fpm===
less /opt/uberspace/etc/$USER/php-fpm.conf

==Python FastAPI==
see https://lab.uberspace.de/guide_fastapi/

# var 1: venv
python3.11 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn
pip install fastapi.responses
pip install gunicorn uvloop httptools
deactivate

# var 2: not use venv
pip3.11 install --user fastapi uvicorn fastapi.responses gunicorn uvloop httptools

# config backend
uberspace web backend set /strava-be --http --port 9001
# --remove-prefix
uberspace web backend list

supervisorctl restart fastapi

~/fastapi/conf.py
#!/usr/bin/env python3.10

# apply changes via
# supervisorctl restart fastapi

import os

app_path = os.environ["HOME"] + "/fastapi"

# Gunicorn configuration
wsgi_app = "main:api"
bind = ":9001"
chdir = app_path
workers = 1
worker_class = "uvicorn.workers.UvicornWorker"

# errorlog = app_path + "/errors.log"
# accesslog = app_path + "/access.log"

~/fastapi/main.py
#!/usr/bin/env python3.10

# apply changes via
# supervisorctl restart fastapi

from fastapi import FastAPI
from fastapi.responses import JSONResponse
from pydantic import BaseModel

import subprocess

class StravaSession(BaseModel):
sessionId: str

api = FastAPI()

@api.post("/strava-be/activityStats2/")
async def activityStats2(session: StravaSession):
# return {session}
response = {}
# response = {"session": session.sessionId}
process = subprocess.run(
[
"python3.10",
"/var/www/virtual/entorb/html/strava/activityStats2.py",
session.sessionId,
],
capture_output=True,
)

if process.returncode == 0:
response["status"] = "ok"
response_code = 200
# response["process_stdout"] = process.stdout.decode()
else:
response["status"] = "error"
response_code = 400
# response["process_stdout"] = process.stdout.decode()
response["error_message"] = process.stderr.decode()
return JSONResponse(content=response, status_code=response_code)

# To test this app locally, uncomment:
# import uvicorn
# uvicorn.run(api, host="localhost", port=8001)

# curl -i -X POST "https://entorb.net/strava-be/activityStats2/" -H "Content-Type: application/json" -d '{"sessionId": "1234"}'

==Checkliste neuer U7 Account==
~/.bashrc und ~/.bash_profile von altem angeglichen
Logs aktiviert
uberspace web log access enable
uberspace web log apache_error enable
uberspace web log php_error enable
uberspace web errorpage 500 disable

==Probleme bei der Migration Uberspace U6 zu U7==
===0. E-Mails===
Es scheinen keine E-Mails mehr rein und raus zu gehen. (Account xxx@entorb.net).

Neuer Webmailer ist https://webmail.uberspace.de , dieser zeigt aber erst nach Abschluss der Migration auf das neue Postfach. Solange die Migration nicht abgeschlossen ist, muss im Mail Client (wie Thunderbird) dieser Login für den U7 Mail Account verwendet werden:
statt xxx@entorb.net -> xxx@entorb.uber.space

===1. HTTPS Zertifikat===
HTTPS Zertifikat war auf einmal nicht mehr gültig für https://www.entorb.net . Das hatte erst auch auf dem U7 funktioniert, nun plötzlich nicht mehr. UPDATE: Problem verschwand von alleine, hing vielleicht mit dem DNS Umzug zusammen.

===2. bestehende ezmlm Mailinglisten===
ezmlm ist gemäß [https://lab.uberspace.de/guide_ezmlm.html Anleitung] installiert, allerdings sagt mir
ezmlm-make -+ ~/ezmlm/oldlist ~/.qmail-oldlist oldlist entorb.uber.space
> ezmlm-make: fatal: unable to stat /etc/ezmlm/de: file does not exist

Wenn ich hingegen testweise einen neuen Verteiler anlege tut es:
ezmlm-make -A -u -m -5 xxx@entorb.net ~/ezmlm/mylist ~/.qmail-mylist
mylist entorb.uber.space
und diesen kann ich dann auch bearbeiten via
ezmlm-make -+ ~/ezmlm/mylist ~/.qmail-mylist mylist entorb.uber.space

Lösung:
in ezmlm/oldlist/ezmlmrc die entsprechende Zeile löschen

===3. Zugriff auf Dateien außerhab von ~/html===
Ich hatte einige Dateien (wie zB eine sqlite DB) in meinem Home (~/) liegen. Auf diese sollen Skripte die sich unter
~/html->/var/www/virtual/entorb/html
befinden zugreifen können. Das funktionierte nicht mehr, da SELinux hier rein grätscht

Beste Lösung: Die Dateien auf die der httpd zugreifen soll nach
/var/www/virtual/entorb/
verschieben und im Home Symlink hinterlassen.

Erster Ansatz: der SELinux Kontexte lässt sich so setzen (Siehe [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-security-enhanced_linux-working_with_selinux-selinux_contexts_labeling_files]
).
chcon -t httpd_sys_content_t file-name
half hier aber nicht:
ls -laZ /home/ |grep ento
drwx------. entorb entorb unconfined_u:object_r:user_home_dir_t:s0 entorb

===4. Perl Module, wie zB Excel::Writer::XLSX===
Installiert via
cpanm Excel::Writer::XLSX

Führe ich das Skript ~/html/test.pl in der Shell aus, wird das Modul
gefunden und geladen. Im Browser via https://entorb.net/test.pl wird
das Modul nicht gefunden.

Ursache 4.1 war dass SELinux den Zugriff des httpd auf /home/entorb/perl5 blockierte

Lösung:
~/perl5 Verzeichnis nach
/var/www/virtual/entorb/perl5
verschoben und im Home einen Symlink hinterlassen.

Dann zuerst lib::local installiert, dann andere Module.

Ursache 4.2 war dass ich in der .bashrc dies gesetzt habe:
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
das musste ich im Perl Skript ebenfalls hinterlegen:
use lib ('/var/www/virtual/entorb/perl5/lib/perl5');
# aber !kein! use local::lib; !!!

===5. sendmail===
ich stellte fest, dass ich im U7 aus Python oder Perl CGI Skripten nicht mehr auf sendmail zugreifen kann. Über die Shell kommt die Mail sofort an. Beispiel:
8<---
import os
SENDMAIL = "/usr/sbin/sendmail"
to = "xxx@entorb.net"
subject = "testmail"
sender = "U7 <no-reply@entorb.net>"
body = "leer"
mail = f"To: {to}\nSubject: {subject}\nFrom: {sender}\nContent-Type:
text/plain; charset=\"utf-8\"\n\n{body}"
p = os.popen(f"{SENDMAIL} -t -i", "w")
p.write(mail)
p.close()
8<---
Lösung: statt Versand aus Web-Skripten/Seiten via sendmail, besser aus den Skripten ein Insert in eine SQLite DB machen und diese via cronjob und php mail() Funktion periodisch abarbeiten.

===6. Wiederholung der Migration===
Entweder einfach Migration auf dem U6 nochmal starten via
uberspace-move-account -u entorb
oder in Dashboard den Umzug abbrechen und neu starten -> anderer Server.

===7. MySQL DB Passwort angeblich zu kurz===
Das Migrationsskript hatte behauptet, dass mein MySQL Passwort zu kurz sei. Stimmte nicht, vermutlich kam es mit den Sonderzeichen durcheinander. Habe daher ein neues langes ohne Sonderzeichen vergeben.

==Streamlit==
see also [https://github.com/entorb/strava-streamlit/blob/main/README.md strava-streamlit readme]

Installation
pip3.11 install --user streamlit

Setup dir and config
mkdir ~/strava-streamlit
cd ~/strava-streamlit
mkdir .streamlit
vim .streamlit/config.toml

[browser]
gatherUsageStats = false

[server]
headless = true
port = 8501
baseUrlPath = "/strava-streamlit"

baseUrlPath is important when not running in web server document root

create minimal Streamlit app
vim src/app.py

import streamlit as st
st.title("Test")

start it manually as testrun (stop by ctrl+c)
streamlit run src/app.py

Add web backend
uberspace web backend set /strava-streamlit --http --port 8501
maybe add --remove-prefix

Access it via browser
https://entorb.net/strava-streamlit

Create Service
vim ~/etc/services.d/strava-streamlit.ini

[program:strava-streamlit]
directory=%(ENV_HOME)s/strava-streamlit
command=streamlit run src/app.py
# or
# command=python3.11 -O -m streamlit run src/app.py
# for env var:
environment=MY_VAR="asdf"
loglevel=info

Start Service
supervisorctl reread
supervisorctl update
supervisorctl status
supervisorctl restart strava-streamlit

Check log
supervisorctl tail -f strava-streamlit
# as this is empty:
tail -f ~/logs/supervisord.log

Apache

2026-03-17T05:31:30Z

Torben: /* Token Query Parameter protection */

[[Category:Webserver]]
==.htaccess==
===basics===
# a different .htaccess file can be stored in each subfolder

# set the timezone
SetEnv TZ Europe/Berlin

# do not generate a index.htm listing of the files
Options -Indexes
# can be turned on for a specifiv subfolder by using another .htaccess
# in that folder containing "Options Indexes" without the "-"

# disable the display on index-pages for files matching * = all
IndexIgnore *.jpg

# ensure that http://entorb.net --> http://www.entorb.net
RewriteCond %{HTTP_HOST} !^www\.entorb\.net$
RewriteRule ^(.*)$ http://www.entorb.net/$1 [L,R=301]

===Password===
Ask for a password when accessing a folder

[http://httpd.apache.org/docs/1.3/howto/htaccess.html]
Create a new passwd file
htpasswd -c -m /some/path/passwd.file myUser
Add another user
htpasswd -m /some/path/passwd.file myUser2

( at Uberspace the <passwd.file> has to be located below /var/www/virtual/<USER>/ )

.htaccess:
AuthType Basic
AuthName "some Text"
AuthUserFile /some/path/passwd.file
AuthBasicProvider file
Require valid-user

===Token Query Parameter protection===
# require a token query parameter to be set
RewriteEngine On
RewriteCond %{QUERY_STRING} !token=asdf
RewriteRule ^ - [F]

===URL Rewrite for wiki===
see
[http://www.stephan-hertz.de/blog/modrewrite-und-suchmaschinen-optimierung-fur-ein-wiki-via-htaccess/]
, [http://www.mediawiki.org/wiki/Manual:Short_URL/wiki/Page_title_--_DreamHost_Shared_Hosting]
or [http://www.mediawiki.org/wiki/Manual:Short_URL/wiki/Page_title_--_no_root_access]

my case: wiki stored unter folder wiki, using virtual folder wickie
# Wiki: convert request using /index.php?title=XYZ ->/XYZ
# only apply the following to wiki urls
RewriteCond %{REQUEST_URI} .*(wickie|wiki).* [nocase]
RewriteCond %{THE_REQUEST} \?title=(.+)\ HTTP
RewriteRule ^.*$ http://www\.entorb\.net/wickie/%1? [R=301,L]

# Short URLs for Wiki
#http://www.mediawiki.org/wiki/Manual:Short_URL/wiki/Page_title_--_DreamHost_Shared_Hosting
RewriteEngine On
# only apply the following to wiki urls
RewriteCond %{REQUEST_URI} .*(wickie|wiki).* [nocase]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)$ /wiki/index.php?title=$1 [L,QSA]
# in LocalSettings.php
$wgUsePathInfo = true;
$wgScriptPath = "/wiki"; # real path
$wgScriptExtension = ".php";
$wgScript = "$wgScriptPath/index$wgScriptExtension";
$wgArticlePath = "/wickie/$1"; # virtual path

==robots.txt==
Place in webservers-root-dir. To keep search robots out of some of your folders. Attention: this file my be a hint for intruders...
User-agent: *
Disallow: /privateStuff/
Disallow: /Gallery/

Pandas

2026-03-08T10:30:38Z

Torben: /* Basics */

[[Category:Coding]][[Category:Python]]
==Getting started==
import pandas as pd

===Docu===
nice cheat sheets can be found here
* https://www.datacamp.com/cheat-sheet/pandas-cheat-sheet-for-data-science-in-python
* https://www.datacamp.com/cheat-sheet/pandas-cheat-sheet-data-wrangling-in-python

==Create DF==
import pandas as pd
# empty
df = pd.DataFrame()
===random data===
df = pd.DataFrame({"x": range(100)})
df["y"] = np.random.rand(len(df)) * 10

# y = sin(x) + noise
noise = np.random.rand(len(df)) * 0.1
noise = np.random.default_rng().random(10) # newer syntax
df["y"] = np.sin(df["x"] / 25 * np.pi) + noise

# 20 rows and 3 columns of random values
df = pd.DataFrame(np.random.rand(20, 3), columns=["a", "b", "c"])

===from Lists===

# from 1 dim list
df = pd.DataFrame(data={"Column Name": lst})
# per column
df = pd.DataFrame(
data={"col1":lst1, "col2":lst2, "col3":lst3})
)
# from N dim list, and set name of columns
df = pd.DataFrame(
data=lst, columns=("col1", "col2", "col3)
)
# from multiple lists
data = zip(
l_days,
l_2016,
l_2017,
l_2018,
l_2019,
l_2020,
strict=True,
)
...

===from csv===
see [https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html]
df = pd.read_csv("data.tsv", sep="\t")
# only selected columns and parse date
df = pd.read_csv(
"data.tsv",
sep="\t",
decimal=",",
usecols=[
"Date",
"Deaths_New",
], # only load these columns
parse_dates=[
"Date",
], # convert to date object if format is yyyy-mm-dd pr dd.mm.yyyy
index_col="Date", # choose this column as index
na_values=["<4"], # values to treat as NA
)
df = df.rename(columns={"Deaths_New": "Deaths_Covid"})

# convert data upon loading
# read only first 10 chars from 2021-04-29T12:15:00+02:00 -> 2021-04-29
pd_date_converter = lambda x: (x[0:10]) # noqa: E731
df = pd.read_csv(
"data.tsv",
sep=",",
converters={"Datum": pd_date_converter},
parse_dates=[
"Datum", # # convert to date object if format is yyyy-mm-dd
],
)
# convert datetime to date after loading
for c in ("Buchungstag", "Valutadatum"):
df2[c] = pd.to_datetime(df2[c], format="%d.%m.%Y").dt.date

===from list[str] via read_csv===
from io import StringIO

csv_string_io = StringIO("\n".join(lines))
df = pd.read_csv(
csv_string_io,
sep=";",
)

data_str = """
datetime_de value
30.09.2022 10:00 214516.10
13.10.2022 00:20 214563.85
"""
df = pd.read_csv(StringIO(data_str), sep="\t", lineterminator="\n")
df["value"] = df["value"].astype(float)

===from Excel===
There are 2 libs: openpyxl (for read and write) and XlsxWriter (write)
see [https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html]
import openpyxl # pip install openpyxl

df = pd.read_excel("file.xlsx", sheet_name="Sheet1", engine="openpyxl")

df = pd.read_excel(
"file.xlsx",
skiprows=4,
usecols=[
"ArbPlatz",
"BezArt",
],
)

===from dict===
see [https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.from_dict.html]
df = pd.DataFrame.from_dict(d, orient="index", columns=["Col1", Col2])

flatten dict in list of dict
# l1 is list of dict, each dict has a sub-dict called metadata
l2 = []
for item in l1:
flat = item.copy()
meta = item.pop("metadata")
flat.update(meta)
l2.append(flat)
del l1

df = pd.DataFrame.from_dict(l2)

===from array/record/list===
see [https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.from_records.html]
data = [ {"id":1, "name":"asdf"}, {"id":2, "name":"qwertz"}, ]
df = pd.DataFrame.from_records(data)

===from JSON===
see [https://pandas.pydata.org/docs/reference/api/pandas.read_json.html]
df = pd.read_json("file.json")

===from SQLite===
import sqlite3
sql = "SELECT * FROM tab WHERE col2 = ? ORDER BY id";
df = pd.read_sql_query(sql, con, params=(MY_FILTER_VALUE,))

==Export Data==
===to csv===
df.to_csv(
"data.tsv",
sep="\t",
lineterminator="\n",
)
# selected columns only
df[ ["Col1", "Col2"] ].to_csv(
"data.tsv",
sep="\t",
lineterminator="\n",
)
to string in CSV format
l_csv = (
df.drop(columns=["hour"]).to_csv(
index=False, sep="\t", lineterminator="\n", encoding="utf-8"
),
)
s_csv = "".join(l_csv)

===to Excel===
see [https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_excel.html]
df.to_excel("data.xlsx", index=False)

Export 2 df as sheets in 1 file
with pd.ExcelWriter(path=file_in.with_suffix(".xlsx")) as writer:
df1.to_excel(writer, sheet_name='Sheet_name_1')
df2.to_excel(writer, sheet_name='Sheet_name_2')

===to HTML===
see [https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_html.html]
df.to_html(
"out.html", index=False, render_links=False, escape=False, justify="center"
)

# html encoding of column name only
df["name"] = df["name"].str.encode("ascii", "xmlcharrefreplace").str.decode("utf-8")
# add link to name
df["name"] = "<a href='" + df["url"] + "' target='_blank' >" + df["name"] + "</a>"
# export to html
df[ ["name", "due", "overdue", "priority", "overdue priority"] ].to_html(
"out.html", index=False, render_links=False, escape=False, justify="center"
)

===to JSON===
json_data = df.values.tolist()

with Path(pathStatsExport / "ts_types_month.json").open(
"w", encoding="UTF-8"
) as fh:
json.dump(
json_data,
fp=fh,
ensure_ascii=False,
sort_keys=False,
indent=2,
)

==Modification==
===Add Row===
add row from list to end
idx = df.index[-1] + 1
list_of_values = (...)
df.loc[idx] = list_of_values

add dummy row for missing 1.1.2020 and reindex
df.loc[-1] = "2020-01-01", 0
df.index = df.index + 1 # shifting index
df = df.sort_index() # sorting by index

===Add column based on other columns===
df["URL"] = df.apply(
lambda row: f"https://www.openstreetmap.org/?mlat={row['Lat']}&mlon={row['Lng']}#map={zoom}/{row['Lat']}/{row['Lng']}",
axis=1,

===Apply map/dict id->name===
df["x_gear_name"] = df["gear_id"].map(d_id_name)

===Replace na Values===
df = df.dropna() # removes missing values
# only in certain columns
df = df.dropna(subset=["power", "heart_rate", "cadence"])
df = df[df["value"].notna()]

===Interpolate missing values===
# interpolate missing data, e.g. after df.groupby([pd.Grouper(key="time", freq="1min")])
df["kWh_total_in"] = df["kWh_total_in"].interpolate(method="linear")

===Calc Diff / Delta of rows===
df["delta_prev"] = df["kWh"].diff()
df["delta_next"] = df["kWh"].shift(-1) - df["kWh"]

===Overwrite Data===
# overwrite values > 123 by 123 etc.
df["Lat"] = df["Lat"].clip(lower=-180, upper=180)
df["Lng"] = df["Lng"].clip(lower=-90, upper=90 )

# set value of columns based on condition
# set "type" = "Ride" where "type" == "VirtualRide"
df.loc[df["type"] == "VirtualRide", "type"] = "Ride"

# negative -> 0
df[df < 0] = 0

# overwrite column data of last 3 weeks by None
df["DateAsDate"] = pd.to_datetime(df["Date"], format="%Y-%m-%d")
date_3w = dt.date.today() - dt.timedelta(weeks=3)
df.loc[df["DateAsDate"].dt.date >= date_3w, "MyColumn"] = None

# rolling takes NAN values into account, so I need to overwrite them as well
df3["Deaths_Covid_roll"] = np.where(
df3["Deaths_Covid"].isnull(), np.nan, df3["Deaths_Covid_roll"]

df.loc[(df["Kat2"] == "") & (df["IBAN"] == "DE02100100100152517108"), "Kat2"] = "Bahn"
df.loc[(df["Kat1"] == "") & (df["Kat2"] == "Bahn"), "Kat1"] = "Mobilität"

# assigning multiple values
df.loc[
(df["Kat1"] == "") & (df["IBAN"] == "DE02100100100152517108"),
["Kat1", "Kat2", "Kat3"],
] = ["Mobilität], "Reisen", "Bahn"]

# str endswith
df.loc[
(df["RespTime"] == 0.0)
& (df["Text"].str.endswith("(read timeout=30)")),
"RespTime",
] = 30.0

==Read-out Data==
===Columns===
df2 = df [ [ "col1", "col2" ] ]

===Rows===
for index, row in df.iterrows():

===extract cell based on Index===
col1_first = df2["col1"].iloc[0]
col1_last = float(df["col1"].iloc[-1])
de_sum = df["col"].loc["Summe"]

===Loop over DF===
for row in df.itertuples():
assert type(row.start_date) is dt.datetime
assert type(row.utc_offset) is int
start_date = row.start_date - dt.timedelta(seconds=row.utc_offset)

==Aggregation==
===Sum Column or Row===
# sum of 1 column
sum_cases = df["Cases"].sum()
# sum per column
df_sums = df.sum(axis="columns")
# sum per row / index
df_sums = df.sum(axis="index")

==Filtering==
===Basics===
# new notation
var = 40
df = df.query("power >= 80 & cadence >= @var")

# old notation
df = df[df["x_start_locality"].isna() & df["start_latlng"].notna()]

# Remove some cat
df = df[~df["Kategorie"].isin(KAT_REMOVE)]

# Remove missing date
df = df[df["Datum"].notna()]

df = df[ df["power"] >= 80 ]
df = df[ (df["power"] >= 80) & (df["cadence"] >= 40) ]
# ~ inverts
# list of multiple values
df = df[ ~df["col1"].isin(("A", "B", "C")) ]

===filter on index===
df = df[df.index >= start_yearweek]

df = df.drop("Summe")

===Filtering Dates===
# filter a date column via year
df = df[df["Date"].dt.year == 2020]
# filter a date via str
df = df[df["Date"] == "2021-11-13"]
# filter on date via dt
df = df[
df["RECEIVE_DATE"].dt.date # convert pandas datetime64[ns] to date
>= (dt.date.today() - dt.timedelta(days=30))
]

===Drop Duplicate Rows===
df = df.drop_duplicates()

===Distinct values of column as list===
gear_ids = df["gear_id"].unique()

==Date Handling==
===Dates===
====Create Date Range====
df = pd.DataFrame({"date": pd.date_range(start="2023-01", end="2024-12", freq="D")})
df = pd.DataFrame({"date": pd.date_range(start="2023-01", end="2024-12", freq="W")})

====Convert Str Column to Date====
# str to datetime
df["Date"] = pd.to_datetime(df["Date"], format="%Y-%m-%d")
# datetime to date
df["Date"] = df["Date"].dt.date

====Extract: year, month, quarter, week, weekstart====
df["year"] = df["date"].dt.year
df["month"] = df["date"].dt.month
df["quarter"] = df["date"].dt.quarter
df["week"] = df["date"].dt.isocalendar().week
df["weekyear"] = df["date"].dt.isocalendar().year
# fix week 53 (when not using weekyear)
# df["x_week"] = df["x_week"].clip(upper=52)
df.loc[(df["week"] == 53) & (df["month"] == 1), "x_week"] = 1
df.loc[(df["week"] == 53) & (df["month"] == 12), "x_week"] = 52

# week_start from date
df["week_start"] = df["date"].dt.to_period("W").apply(lambda r: r.start_time)
# week_start from week number
df["week_start"] = pd.to_datetime(
df.apply(lambda row: dt.date.fromisocalendar(row["year"], row["week"], 1), axis=1)
)

===DateTime===
====Convert String to Datetime and Timezone Handling====
# string to datetime, converting to UTC for future compatibility
df["date"] = pd.to_datetime(df["date"], format="ISO8601", utc=True)
# convert from utc to local timezone
df["date"] = df["date"].dt.tz_convert(tz="Europe/Berlin")
# drop timezone info, since Excel can not handle it
df["date"] = df["date"].dt.tz_localize(None)

====DateTime: remove miliseconds====
df["DateTime"] = df["DateTime"].dt.ceil(freq="s")

====DateTime: extract date and add offset, depending on time====
# values before 15:00 shall be treated end of last day value
df["day"] = df["dt"].dt.date
df.loc[df["dt"].dt.hour < 15, "day"] = df["dt"].dt.date - pd.Timedelta(days=1) # noqa: PLR2004

===TimeStamps===
===Convert Timestamp to Datetime and use as Index===
df["datetime"] = pd.to_datetime( # convert to datetime
df["start_ts"],
unit="s", # timestamp is in seconds
utc=True, # timestamp is in UTC
).dt.tz_convert( # convert to local TZ
"Europe/Berlin"
)
df = df.set_index("datetime")

====Convert Datetime to Timestamp====
df["timestamp"] = df["datetime"].values.astype(np.int64) // 10**9
# or
df["timestamp"] = df["datetime"].astype(int) // 10**9
# // is integer division

====Timestamp: substract first====
# calc elapsed time
df["seconds"] = df["timestamp"] - df["timestamp"].iloc[0]

==String Modifications==
===Replace===
df["name"] = df["name"].str.strip() # trim whitespaces
df["time"] = df["time"].str.replace(r"\..*$", "Z", regex=True)

# remove word "Probable"
df["col"] = df["col"].replace(
to_replace=r"^Probable ", value="", regex=True
)

df["Text"] = df["Text"].str.replace(
r"^Menke, *Torben *",
"Torben Menke",
regex=True,
)

===split string===
df [ [ "week", "year" ] ] = df["Kalenderwoche"].str.split("/", expand=True)

===string <-> number===
string to float: 1.234,56 -> 1234.56
df["Euro"] = (
df["Euro"]
.str.replace(".", "")
.str.replace(",", ".")
.astype(float)
)

convert int to str adding leading zeros
df["Sterbewoche"].astype(str).str.zfill(2)

capitalization
df["Wer"] = df["Wer"].apply(lambda x: string.capwords(x))

trim spaces
df["Wer"] = df["Wer"].str.strip()

===count word per row and check if more than one===
str.count()
df["cnt_Buchungstext"] = df["Text"].str.count("word")
df_search = df[df["cnt_Buchungstext"] != 1]
if len(df_search) > 0:
print(df_search)
del df_search
df = df.drop(columns=["cnt_Buchungstext"])

==Sorting / order by==
# sort by column
df = df.sort_values(by=['betten_belegt'], ascending=False)
df = df.sort_values(by=["count", "query"], ascending=[False, True])
df = df.sort_values(by=["Buchungstag", "Text"], ignore_index=True)

===Top10===
df_top_ten = (
df.sort_values(by="count", ascending=False)
.head(10)
)

==Group By / Top10==
df2 = (
df.groupby("sender")
.size()
.to_frame(name="count")
.sort_values("count", ascending=False)
)

df2 = (
df[ [ "name", "amount" ] ]
.groupby("name")
.agg(amountSum=("amount", "sum"), count=("name", "count"))
.sort_values(by=["amountSum", "count"], ascending=False)
)
print(df2.head(10)) # top10

# resort by count
print(df2.sort_values(by="count", ascending=False).head(10))

===Group by year, quarter, month, week using count and add missing===
Simple: add missing years for multi index
df1 = pd.DataFrame(
columns=["year", "type", "count"],
data=[
(2000, "Run", 1),
(2003, "Ride", 2),
(2003, "Swim", 1),
],
)
df2 = pd.DataFrame(
data={
"year": range(df1["year"].min(), df1["year"].max() + 1),
"type": "Run",
"count": 0,
}
).set_index(["year", "type"])
df1 = df1.set_index(["year", "type"])
df = df1.add(df2, fill_value=0).reset_index()

Advanced
# df has columns: date and derived year, quarter, month, week, see above
# group by year, quarter, month, week, with count as aggregation and gap-filling
year_min, year_max = df["year"].min, df["year"].max
# year
df_year = (
df[ [ "year", "date" ] ].groupby(["year"]).count().rename(columns={"date": "count"})
)
df_year = df_year.reindex(range(year_min(), year_max() + 1), fill_value=None)
df_year = df_year.reset_index().rename(columns={"year": "date-grouping"})

# quarter
df_quarter = (
df[ ["year", "quarter", "date"] ]
.groupby(["year", "quarter"])
.count()
.rename(columns={"date": "count"})
)
df_quarter = df_quarter.reindex(
pd.MultiIndex.from_product(
[range(year_min(), year_max() + 1), range(1, 5)], names=["year", "quarter"]
),
fill_value=None,
)
df_quarter = df_quarter.reset_index()
df_quarter["date-grouping"] = (
df_quarter["year"].astype(str) + "-Q" + df_quarter["quarter"].astype(str)
)
df_quarter = df_quarter.drop(columns=["year", "quarter"])

# month
df_month = (
df[ ["year", "month", "date"] ]
.groupby(["year", "month"])
.count()
.rename(columns={"date": "count"})
)
df_month = df_month.reindex(
pd.MultiIndex.from_product(
[range(year_min(), year_max() + 1), range(1, 13)], names=["year", "month"]
),
fill_value=None,
)
df_month = df_month.reset_index()
df_month["date-grouping"] = (
df_month["year"].astype(str) + "-" + df_month["month"].astype(str).str.zfill(2)
)
df_month = df_month.drop(columns=["year", "month"])

# week
df_week = (
df[ ["year", "week", "date"] ]
.groupby(["year", "week"])
.count()
.rename(columns={"date": "count"})
)
df_week = df_week.reindex(
pd.MultiIndex.from_product(
[range(year_min(), year_max() + 1), range(1, 53)], names=["year", "week"]
),
fill_value=None,
)
df_week = df_week.reset_index()
df_week["date-grouping"] = (
df_week["year"].astype(str) + "-" + df_week["week"].astype(str).str.zfill(2)
)
df_week = df_week.drop(columns=["year", "week"])

===Group by Type and Date===
# group by hour
# truncate min and sec data
df["hour"] = pd.to_datetime(df["time"]).dt.floor("H")
# sum per hour
df2 = df[ ["hour", "amount"] ].groupby("hour").agg(amountSum=("amount", "sum"))
df2.plot()
plt.show()

# Datetime Index Grouping
df = df.groupby(pd.Grouper(freq="5min", offset="00h00min")).max()

# Group by type and date of month start
df_month = df.groupby(["type", pd.Grouper(key="date", freq="MS")]).agg(
{"id": "count", "minutes": "sum"}
)

# generate index of the desired month-freq:
idx = pd.date_range(
start=df["date"].min().replace(day=1),
end=df["date"].max().replace(day=1),
freq="MS", # MS = Month Start
)

# add missing months
df_month = df_month.reindex(
pd.MultiIndex.from_product(
[df_month.index.get_level_values("type"), idx],
names=["type", "date"],
)
)

# fill missing by 0 and convert count back to type int
df_month = df_month.fillna(0).astype({"count": int})

====Group Datetime by Time of Day only====
df["Time"] = df["Date"].dt.round("5min").dt.time

df_grouped = df [ ["RespTime", "Time" ] ].groupby("Time").mean()

====Group and calculate min,avg,max====
df_grouped = (
df[ ["Time", "RespTime"] ]
.groupby("Time")
.agg(
max=pd.NamedAgg(column="RespTime", aggfunc="max"),
avg=pd.NamedAgg(column="RespTime", aggfunc="mean"),
min=pd.NamedAgg(column="RespTime", aggfunc="min"),
)
)

==Column Handling==
===Renaming===
df = df.rename(
columns = {"invasiv_beatmet": "beatmet",},
errors="raise",
)

rename column headers by extracting some int values from a string
l2 = []
for col in df.columns:
year = int(col[0:4])
week = int(col[5:7])
l2.append(year * 100 + week)
df.columns = l2

===Dropping===
drop columns
df = df.drop(columns=["Sterbejahr", "Sterbewoche"])

===Change Column Order===
df = df.reindex(
[
"Wann",
"Art",
"Wer",
],
axis="columns",

# column reorder
cols_first = ["year", "month", "week", "weekyear"]
cols_last = [col for col in df.columns if col not in cols_first]
cols_first.extend(cols_last)
df = df[cols_first]
del cols_first, cols_last

# same but different
first_columns = ("description", "isin", "side", "quantity", "Price", "amount")
i = 0
for col in first_columns:
df.insert(i, col, df.pop(col))
i += 1
del i

==Index Handling==
select column as index
df = df.set_index("Date")
move date from index back to column
df = df.reset_index()

rename index
df.index.name = "Date"

reset index to start at 0
df2 = df1[1 * 365 : 2 * 365].reset_index(drop=True)

int index to str index
df.index = df.index.map(str)

===text indexes===
df = df.set_index("Altersgruppe")

===datetime indexes===
assert type(df.index) is pd.DatetimeIndex
df.index = df.index.tz_convert("Europe/Berlin")
df.index = df.index.tz_convert(None) # to remove tz info

# select "Date" column as index
df = df.set_index(["Date"])

# convert index to datetime
df.index = pd.to_datetime(df.index)

# add missing dates
df = df.reindex(
pd.date_range(df.index.min(), df.index.max(), freq="D"), fill_value=0
)
df.index.name = "date"

# remove timezone offset (for Excel can not handle this)
df.index = df.index.tz_localize(None)

date_last = df.index[-1])

# reindex and fill missing with 0
date_last = pd.to_datetime(df.index[-1]).date()
idx = pd.date_range('2020-01-01', date_last))
df = df.reindex(idx, fill_value=0)

# add missing dates
df = df.asfreq('D', fill_value=0)
df = df.asfreq(freq="5M")
df = df.sort_index() # needed?

# drop values of column for last 3 weeks
date_3w = dt.date.today() - dt.timedelta(weeks=3)
df.loc[df.index.date >= date_3w, "Cases"] = None
# or
df.loc[df.index.date < pd.to_datetime("2020-03-01"), "Value"] = None

====filter on date index====
# drop data prior to 2020
df = df.loc['2020-01-01':]
# alternative:
df = df[df.index >= "2021-01-10"]

==Pivot, UnPivot/Melt==
===Pivot===
# df has columns "key" (column that shall be the index) , "year" (column you want to convert to multiple columns), "value" (the numbers to aggregate)
# convert to wide format with id as columns
df2 = df.pivot_table(index="key", columns="year", values="value", aggfunc="first") # aggfunc="sum"

===melt/unpivot wide table to long table format===
df2 = pd.melt(df, id_vars="Date", value_vars=df.columns[1:]) #
df2 = df2.rename(columns={"variable": "Machine", "value": "Status"})

# not working, using melt instead
# df3 = pd.wide_to_long(df, stubnames="Date", i=df.columns[1:], j="Status")

==Merge/Concat and Append DFs==
===Merge/Concat===
df_sum = pd.DataFrame()
df_sum = pd.concat(
[df_sum, df],
ignore_index=True,
)
===Append===
# merge columns from different dfs, when have matching index, e.g. user_id
df = pd.concat([df1, df2], axis=1)

# append df2 to end of df1
df1 = pd.DataFrame({"date": pd.date_range(start="2020-01", end="2020-02", freq="W")})
df2 = pd.DataFrame({"date": pd.date_range(start="2024-01", end="2024-12", freq="D")})
df = pd.concat([df1, df2]).reset_index(drop=True)

# join series of 2 df
df_covid_2020 = pd.DataFrame()
df_covid_2020['Deaths_Covid_2020'] = df1['Deaths_Covid_2020'].append(
df2['Deaths_Covid_2020'], ignore_index=True)

# note: this would require all index of df in df2 present:
# df[machine] = df2[machine]

==Rolling Average, mean of columns, min, max==
df['2016_roll'] = df['2016'].rolling(window=7, min_periods=1).mean().round(1)
df['2017_roll'] = df['2017'].rolling(window=7, min_periods=1).mean().round(1)
df['2018_roll'] = df['2018'].rolling(window=7, min_periods=1).mean().round(1)
df['2019_roll'] = df['2019'].rolling(window=7, min_periods=1).mean().round(1)
df['2020_roll'] = df['2020'].rolling(window=7, min_periods=1).mean().round(1)
# mean value of 4 columns
df['2016_2019_mean'] = df.iloc[:, [1, 2, 3, 4]
].mean(axis=1) # not column 0 = day
df['2016_2019_mean_roll'] = df['2016_2019_mean'].rolling(
window=7, min_periods=1).mean().round(1)

df['2016_2019_roll_max'] = df.iloc[:, [6, 7, 8, 9]].max(axis=1)
df['2016_2019_roll_min'] = df.iloc[:, [6, 7, 8, 9]].min(axis=1)

==Helpers==
def pandas_set_date_index(df, date_column: str):
""" use date as index """
df[date_column] = pd.to_datetime(df[date_column], format='%Y-%m-%d')
df = df.set_index([date_column])
return df

def pandas_calc_roll_av(df, column: str, days: int = 7):
""" calc rolling average over column """
df[column + '_roll_av'] = df[column].rolling(
window=days, min_periods=1).mean().round(1)
return df

# custom rounding
def custom_round(x: float, base: int = 5) -> int:
"""Custom rounding."""
# from https://stackoverflow.com/questions/40372030/pandas-round-to-the-nearest-n
return int(base * round(float(x) / base))

df["power_rounded"] = df["power"].apply( lambda x: custom_round(x, base=20) )

==Transpose==
df = df.transpose()

==Leftovers==
# when in a function one might get the SettingWithCopyWarning, fix via
df = df.copy()

# copy
df2['Date'] = df0['Date']

# drop 2 rows from the beginning
df = df2.drop([0, 1])

====append today using yesterdays value====
str_today = dt.datetime.today().strftime("%Y-%m-%d")
ts_today = pd.Timestamp(str_today)
if df.index[-1] != ts_today:
df.loc[ts_today] = df.loc[df.index[-1], "Count"]

==Plotting via Matplotlib==
see [[Matplotlib]]

Windows Batch Scripting

2026-03-06T14:27:13Z

Torben: /* Net use for network drive mounting */

[[Category:Software]][[Category:Windows]]
[http://weblogs.asp.net/jongalloway/top-10-dos-batch-tips-yes-dos-batch see "Top 10 DOS Batch tips"] and [http://www.axel-hahn.de/batch/batchecke/tipps/ BATch-Dateien - kleine Tipps] for more stuff

===current dir and filename===
# current working dir
echo %CD%
# path of current file
echo %0
# filename without the extension.
echo %~n0
# filename and extension.
echo %~nx0
# or
echo %~n0%~x0

===Title to Dirname===
@echo off
for %%f in (%cd%) do set dirname=%%~nxf
title %dirname%

===Empty c:\tmp===
@echo off
rmdir /q /s c:\tmp
# or rd /q /s c:\tmp
mkdir c:\tmp

===find files / search (and delete) old files===
via forfiles
forfiles /P "C:\tmp\test-del" /S /D -30 /C "cmd /c echo @path"
forfiles /P "C:\tmp\test-del" /S /D -30 /C "cmd /c del @path"
forfiles /P "C:\tmp\test-del" -S -D 90 -m *.log -c "cmd /c del @path"

via robocopy
robocopy "C:\tmp\test-del" "*.log" "C:\cleanup-logs" /mov /minage:90 /NP
del C:\cleanup-logs /q

===ReadOnly Flag / WriteProtection===
Remove
attrib -h -r -s /s /d *.*
Set
attrib +r c:\Users\torben\Desktop\SyncedFolder\*.* /s

R Read-only file attribute.
S System file attribute.
H Hidden file attribute.
/S Processes matching files in the current folder and all subfolders.
/D Processes folders as well.

===Set Current Working Dir to Script Location ===
This is important if the script is started using a taskmanager etc.
e:
cd %~dp0

===Set Window Title===
TITLE My Window Title
or when opening via START from another Batch file
START "My Window Title" 2.cmd

===Get FolderName===
This when excecuted in c:\sub\folder it returns "folder"
for %%* in (%CD%) do set CurrDirName=%%~nx*
echo %CurrDirName%

===List of files to Textfile===
dir *.* /b > ..\liste.txt

===Pipe output to textfile===
From [https://support.microsoft.com/en-us/help/110930/redirecting-error-messages-from-command-prompt-stderr-stdout] You can print the errors and standard output to a single file by using the "&1" command to redirect the output for STDERR to STDOUT and then sending the output from STDOUT to a file:
some_command > output.log 2>&1
some_command 1> output.log 2> error.log

===exit if command fails with error===
some_command1
IF %ERRORLEVEL% NEQ 0 exit /b %ERRORLEVEL%
some_command2
IF %ERRORLEVEL% NEQ 0 exit /b %ERRORLEVEL%

===Net use for network drive mounting===
net use Y: \\server\path /user:myuser

net use Y: /delete

===Zip */*.ini===
@echo off
setlocal EnableDelayedExpansion
set "OUT=inis.zip"
REM Use PowerShell to expand glob pattern and pipe relative paths to tar with --files-from=-
powershell -NoLogo -NoProfile -Command "Get-ChildItem -Path '*\*.ini' -File | ForEach-Object { $_.FullName.Substring((Get-Location).Path.Length + 1) } | tar -c -a -f '%OUT%' --files-from=-"

===Loops===
====if====
check if mounting (net use) was successful
net use q: \\server\share
q:
cd \
IF NOT EXIST dir1\dir2 (
echo mouting of share failed
pause
c:
net use q: /delete
EXIT /B 1
)

====for====
loop over values
for %%L in (cn, de, en, es, fr, ko, sk, vn) do (
echo %%L
)

loop over all dirs
FOR /D %%D in ("*") DO (
echo %%D
)
double loop over all dirs
FOR /D %%P in ("*") DO (
echo parent = %%P
FOR /D %%C in ("%%P\*") DO (
echo child = %%C
)
)

loop over files
for %%F in (*.pdf) do (
echo %%F
copy %%F %%~nF.old
)

====for and grep.exe====
using UnixUtils grep.exe one can easily extract exceptions from logs:
for %%F in (*.log) do (
grep -B3 -A3 "exception" %%F > %%~nF-error.txt
)

===Variables===
:: no spaces around '='!!!
set xyz=myfile.bat
set /p xyz=Variable Eingeben:
set /p xyz= < TMP.dat
echo %xyz%

====Substrings====
Substring via ":~"
set year=%date:~-4,4
Path, filename and extension of file stored in variable %%F
set path=%%~pF
set name=%%~nF
set ext=%%~cF

====Set Variable to Program Output====
for /f "delims=" %%a in (' powershell -c "$lastmonth = (Get-Date).addMonths(-3); $lastmonth.tostring(\"yyyy-MM\")" ') do set "MONTH=%%a"
echo %MONTH%

===Date & Time ===
====DateString====
set DATESTR=%date:~-4,4%-%date:~-7,2%-%date:~-10,2%_%time:~0,2%:%time:~3,2%:%time:~6,2%
or
set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%%time:~6,2%
:: replace ' ' in small hours with 0
set DATESTR=%DATESTR: =0%
= 240419_085724
zip.exe -9 %DATESTR%.zip *.bat
for better example of zipping see [[Backup#Zip_Folder|zip folder in Backup section]]

====DayOfWeek====
from [https://stackoverflow.com/questions/25537313/issue-with-getting-batch-script-to-print-keeps-given-a-error-about-set-was-unexp here]
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get dayofweek') DO (
if "!count!" GTR "0" GOTO next
set dow=%%D
SET /a count+=1
)
:next
echo %dow%

====Yesterday====
from [https://stackoverflow.com/questions/2954359/dos-batch-programming-howto-get-and-display-yesterday-date here]
set befehl="PowerShell $date = Get-Date; $date=$date.AddDays(-1); $date.ToString('yyyy-MM-dd')"
for /f %%i in ('%befehl%') do set yesterday=%%i

===Copy all files from subfolders to one folder ===
from [https://stackoverflow.com/questions/11720681/windows-batch-copy-files-from-subfolders-to-one-folder here]
set source=c:\source
set target=c:\target
cd %source%
for /r %%a in (*.*) do (
COPY "%%a" "%target%"
)

===FTP===
====FTP-Upload====
file "ftp.bat"
@echo off
ftp "-s:FtpScript"
pause
cls

file "FtpScript"
open www.xyz.de
[User]
[Password]
BINARY
put [File]
quit

===UnixUtils===
Using [https://sourceforge.net/projects/unxutils/ UnixUtils] for Windows wget, grep, etc are usable to scripts in Windows. In the UnixUtils.zip the .exe files are located in foltder usr/local/wbin/ . Examples:
head.exe -c 1000 mylog.log > outHead1000Bytes.log
head.exe -n 1000 mylog.log > outHead1000Lines.log
tail.exe -c 1000 mylog.log > outTail1000Bytes.log
tail.exe -n 1000 mylog.log > outTail1000Lines.log
grep.exe -i -B3 -A1 "ERROR" mylog.log > outGrepErrors.log

====Curl to check if page online====
@echo off
set URL1=https://my.server.com
:loop
for /f "delims=" %%i in ('curl -o nul -I -s -w "%%{http_code}" "%URL%"') do set HTTP_STATUS=%%i
echo Status: %HTTP_STATUS% at %date% %time%
timeout /t 5
goto loop

===TOP CPU RAM Disk I/O===
# CPU + RAM
top
# I/O
sudo iotop -o -d 2 -k

===Run as other user===
runas /user:MYDOMAIN\myuser /savecred "cmd /K cd c:\mydir && my_programm.exe"

/K to keep window open when finished
/C to close window when finished

AI LLM for Coding

2026-02-22T06:58:50Z

Torben:

Python

2026-02-17T08:26:01Z

Torben: /* Process Bar */

Linux Bash Shell Scripting

2026-01-17T05:26:58Z

Torben: /* Parameters */

[[Category:Linux]]
===Bash commands===
rm -rf dir # do not show warning if not existing
mkdir -p dir # do not print warning if already existing AND make needed parents

===Output===
View File
less File
Print last line of file
tail -1 File
Print last line, column 2 of file (cut expects tab separating)
tail -1 File | cut -f2
Follow changes in (log)file
tail -f File
Print file contents to std output
cat File
Filter contents of file to std output
grep "ERROR" File
Filter contents of file to std output (case-insensitive)
grep -i "error" File
Filter contents of file to std output, +2 lines before +1 following lines each
grep -B2 -A1 "ERROR" File
Print file contents to std output, starting at key word
grep -A9999999 "2016-04-19T10:38" File
Save program output (script/grep/cat...) into file
sh script.sh > log.txt # std output, (errors to terminal)
sh script.sh 1> log.txt 2> log-errors.txt # separate files
sh script.sh &> log.txt # std and errors both to 1 file
Send output to file and terminal
touch log.txt & tail -f log.txt & sh script.sh > log.txt
Piping
sh script.sh | less
Filtering of output
sh script.sh | grep tmenke

====Search file contents in dir recursively====
From [https://stackoverflow.com/posts/16957078/timeline]
grep -rn '/path/to/somewhere/' -e 'pattern'

===Input===
====Enter a value====
echo "Enter to continue, CTRL+C to cancel"
read ok

echo -n "Was? "
read var
echo "$var"

====Dialog Input====
dialog --yesno "Cleanup Old Backups First" 0 0
CleanUpBackups=$?
# yes -> 0 # no -> 1
if [ $CleanUpBackups -eq 0 ] # spaces needed!!!
then
perl /save/_save/backupPlatteAufraemen/loeschen2.pl
fi

===Loops===
Zip all subdirs
for D in `ls -d */`
do
# remove tailing /
D=${D%/}
echo === $D ====
cd $D
zip -rq9 ~/sicher/html/$D.zip .
# tar cfz ~/sicher/html/$D.tgz .
cd ..
done

Check if variable in list
if [[ "$D" =~ ^(hpmor-en/|DukeTypem2D/)$ ]]; then
echo skipping $D

Good overview of parameters for if [http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html]
 If
if [ -d "backup.0" ]; then
echo "gibts"
else
echo "gibts nicht"
fi
one-liner:
if [ -d "backup.0" ] ; then sudo mv backup.0 backup.1 ; fi

Check if process is running
if [ ! "$(pidof workrave)" ] ; then
nohup workrave
fi

While 
Normal loop
i=1
while [ $i -le 4 ] do
(( i++ ))
done
Endless loop
while true; do
# ...
echo "Enter to continue, CTRL+C to cancel"
read user_input
done
For loop
for FILE in *.txt;
do
mv "$FILE" `echo $FILE | tr ' ' '_'`
done

Check if files matching *.xml exist, store Number as string and print it
NUM=$(find $TRANSPORT/*.xml -type f 2> /dev/null |wc -l)
echo $NUM
if [ $NUM != "0" ] ; then
echo "$NUM files found, copying"
else
echo "No files found, exiting"
exit 0
fi

====Loop over contents / columns of file====
file contains tab separated columns $1 -> column1 etc.
while read line ; do
set $line
myCommit=$1
myDate=$2
echo "$myCommit ; $myDate"
done <"history-change.tsv"

===Exit upon Error===
set -e
touch /tmp/not_found.txt
echo will not reach here

===Parameters===
from [http://stackoverflow.com/questions/3534280/how-can-i-pass-a-file-argument-to-my-bash-script-using-a-terminal-command-in-lin]
OPTIONS=$(getopt -o f:t -l file:,titlepage -- "$@")
if [ $? -ne 0 ]; then
echo "getopt error"
exit 1
fi
eval set -- $OPTIONS
while true; do
case "$1" in
-f|--file) FILE="$2" ; shift ;;
-t|--titlepage) TITLEPAGE=1 ;;
--) shift ; break ;;
*) echo "unknown option: $1" ; exit 1 ;;
esac
shift
done
if [ $# -ne 0 ]; then
echo "unknown option(s): $@"
exit 1
fi

if [ ! $FILE ]; then
echo "no file given as parameter"
exit 1
fi
if [ ! -f $FILE ]; then
echo "file '$FILE' not found"
exit 1
fi
# echo "titlepage: $TITLEPAGE"
# echo "file: $FILE"
# exit 1

===Combine find and some other command e.g chmod===
# chmod for dirs
find -type d -exec chmod g+xs {} \;

# make scripts excecuteable
find . -name "*.sh" -exec chmod 744 {} \;

# remove .zip files older 30 days
find ./R*/logs/*.zip -mtime +30 -exec rm {} \;

# Copy empty Directory Structure
mkdir /Zielverzeichnis
cd Quellverzeichnis
find . -type d -depth | cpio -pvdma /Zielverzeichnis/

===sed===
If you want to replace the text 'asdf' by 'qwert' in all '*.txt' files in a directory at once, use the following command:
sed -i -e 's/asdf/qwert/g' *.txt

add a line to file
sed -i '8i\\\input{layout/hp-format}\n\\input{layout/hp-markup}\n' $target_file

delete/"grep out" lines
sed -i '/\\\input{layout\/hp-contents}/d' $target_file

insert file into file at marker
sed -i -e '/<style/r ebook/epub.css' $target_file

# -z changes the delimiter to null characters (\0) to allow for multiline matching
sed -i -z 's/<\/header>.*HARRY POTTER<\/p>/<\/header>\nHARRY POTTER<\/p>/' $target_file

===get filename dirname etc===
cd to parent dir
script_dir=$(dirname $0)
cd $script_dir/..

dirname file
basename file

myPath=data/de-districts/de-district_timeseries-02000.tsv
myDir=$(dirname -- "$myPath")
myFileName=$(basename -- "$myPath")
myFileExt="${myFileName##*.}"
myFileName="${myFileName%.*}"

filename without extension
FILEOLD="${FILE%.pdf}-alt.pdf"

for file in *
do
if [ -f $file ] ; then
name=${file%\.*} # name without extension
echo ${name}
fi ;
done

name=`basename /path/filename.ext .ext` # -> filename
for i in *.jpeg; do mv "$i" "`basename "$i" .jpeg`.jpg"; done

Path of current working dir
dir=$PWD

Name of current directory without path
dirname=${PWD##*/}

Location of this script
dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

===Get Date as String===
DATE=`date +"%y-%m-%d_%H-%M"`
echo $DATE

===Count Files in Folder===
find . | wc -l
Check free inodes on partition
df -i

===Tar===
get rid of message "removing leading /": use option P

===Pause===
read -p "Press any key to start backup..."

===fstab mount discs harddrive partitions===
read UUID
sudo blkid
sudo blkid /dev/sda3

UUID=D4AE21F4AE21D032 /media/win ntfs auto,user,uid=torben,gid=torben,fmask=0177,dmask=0077,ro 0 0

UUID=f103902f-bbd6-49af-b983-b1dddedd361a /media/linux ext3 auto,user,rw 0 0

UUID=EA06-C29C /media/files vfat auto,user,utf8,uid=torben,gid=torben,fmask=0177,dmask=0077,rw 0 0

===Network Tests===
ping 10.10.10.10 -c 1
netcat / ncat / nc 10.10.10.10 8080
telnet 10.10.10.10 8080

====netcat====
Netcat is pretty cool, as it can be used to send messages between two machines, or even set files, see [https://www.digitalocean.com/community/tutorials/how-to-use-netcat-to-establish-and-test-tcp-and-udp-connections-on-a-vps]

# host1: start to listen on a port
netcat -l <PORT>
# host2: connect to host1
netcat <IP-host1> <PORT>
# now messages can be send in both ways
Transfer file
# host1: start to listen on a port
netcat -l <PORT> > received_file
# host2: send a file
netcat <IP-host1> <PORT> < original_file

====wget====
wget --no-check-certificate https://192.168.0.123:8080
wget --no-check-certificate --user=myUser --password=myPasswd https://10.10.10.10/url.xml
# DynDNS
wget -d -v "http://192.168.0.123:8080/nic/update?hostname=myhost.10.10.1.100&myip=10.10.10.12&localip=10.10.10.12" --user="u123456" --password="p123456" -O output.xml
wget --ca-certificate=MyCert.pem --user=myUser --password=myPwd https://myUrl:80/file.xml

====curl====
curl --trace /dev/stdout \
--cacert MyCert.pem \
https://myUrl:80/file.xml \
-u myUser:myPwd \
--header "Content-Type: text/xml;charset=UTF-8" \
--header "SOAPAction: http://MySoapActionURL" \
--header "Accept-Encoding: gzip, deflate" \
--data '<MySoapData>'

Access Strava API and retrieve JSON format
curl -X GET "https://www.strava.com/api/v3/athlete?access_token=1234567890" -H "accept: application/json"

==Examples==
Search for a word in some files (here *.txt)
if [ -z $1 ]; then
echo -n "Was? "
read was
else
was=$1
echo "$was"
fi
grep --color=auto -i $was *.txt

===Crontab===
# every 5 minutes
*/5 * * * * script1.sh
# every 5 minutes + 1
1-56/5 * * * * script2.sh

===systemctl Services===
# create service
sudo vim /etc/systemd/system/mqtt_influx.service

[Unit]
Description=My MQTT to InfluxDB Service

[Service]
ExecStart=/usr/bin/env python3.9 /home/pi/influx-collectors/mqtt.py
WorkingDirectory=/home/pi/influx-collectors
Restart=always
User=pi
Group=pi
Nice=10

[Install]
WantedBy=multi-user.target

# activate and start service
sudo systemctl daemon-reload
sudo systemctl enable mqtt_influx.service
sudo systemctl start mqtt_influx.service
sudo systemctl restart mqtt_influx.service
sudo systemctl stop mqtt_influx.service
sudo systemctl status mqtt_influx.service
# access log (-f = follow)
sudo journalctl -u mqtt_influx.service -f

===Clean Harddisk by filling with Zeros===
Caution: double check which device you are cleaning...
sudo dd if=/dev/zero of=/dev/sdd1 bs=1M

===Screen===
# liste anzeigen
screen -list
# neuen starten
screen -R name
# screen in Hintergrund
STRG+a d -> detact screen
Screen Start Skript in new Screen
screen -A -m -d -S myScrName ./start.sh
kill Screen Session
STRG+d
kill Background Screen Session without entering
screen -X -S myScrName quit

===PID store and kill===
Set the name of the process
bash -c "exec -a <MyName> <Command>"
kill the process by name
pkill -f MyName

get and store PID
/home/user/bin/script.sh & echo $! > script.pid
kill process using PID
kill $(cat script.pid)

===Check if running as root/sudo===
if [[ $EUID -ne 0 ]]; then
echo "E: run via sudo!!!" 2>&1
exit 1
fi

check for username
if [ `whoami` != 'torben' ]
then
echo "You must be torben to do this."
exit
fi

===Add Group and User and Set Password===
groupadd testuser
useradd testuser -g testuser
echo -e "testpass\ntestpass" | (passwd --stdin testuser)

===Monitoring open files of a process===
## check system limit
# ulimit -n
DATE=`date +"%y-%m-%d_%H-%M"`
ID=`pgrep -U username -f ProcessName`
FILES=`ls -1 /proc/$ID/fd/ | wc -l`
echo "$DATE $FILES ">> /home/user/logfile.log
## check process limit
# cat /proc/$ID/limits

===Backup Script for folder using tar and filters===
#!/bin/bash
initialdir=$PWD
DATE=`date +"%Y-%m-%d_%H-%M"`

# cd to "path of this file /.."
dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $dir/..
# name of current folder = name to use for backup
name=${PWD##*/}
backupSource=$PWD

cd $backupSource/../backup/
zipDir=$(pwd)
zipTarget=$zipDir/$name-$DATE.tgz

cd $backupSource
tar cfvz $zipTarget ./ --exclude='logs/*' --exclude='import/*'

cd $initialdir

===Zip dir including hidden files===
zip -r9q myfile.zip . -x '.git/*'

===Start Stop Script for Java Application===
#!/bin/bash
# Start and Stop Linux Service for Java application
# uses java paramerter -Dname=$PROGRAM
# pgrep -f $PROGRAM
# pkill -f $PROGRAM
# with $PROGRAM = Name of directory above location of this script:
# /home/torben/Tool/bin/startscript.sh -> Tool
# STDOUT and STDERR are written into ./logs/$PROGRAM-STDOUT.log

# cd to "path of this file /.." to get dirname to use as service name
dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
cd $dir/..
dirname=${PWD##*/}
# Attention, name has to be unique on the system -> added -Service !!!
PROGRAM=$dirname-Service

# ensure that this script can only be started as user torben and not accidently as root
username=torben

jar="./MyJavaApplication.jar"
log_conf="./config/log4j2.xml"
app_params="-conf ./config/config.xml"

vm_params=" \
-Dname=$PROGRAM \
-Dlog4j.configurationFile=$log_conf \
-DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector \
"
server_params=" \
-server -Xms512m -Xmx2048m \
-XX:+UseParallelGC \
-XX:+AggressiveOpts \
-XX:+UseFastAccessorMethods \
-XX:-OmitStackTraceInFastThrow \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=./logs/rss-heap-$(date +%Y%m%d_%H%M%S).hprof \
"
# for remote jmx access (disabled by default)
jmx_params=" \
-Djava.rmi.server.hostname=172.19.13.200 \
-Dcom.sun.management.jmxremote.port=1100 \
-Dcom.sun.management.jmxremote.local.only=false \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
"
JAVAPROGRAM="\
$vm_params \
-jar $jar $app_params \
$server_params \
"

# for remote jmx access add
# $jmx_params

# for problems with IPv6 add (i.e. if Startup takes more than 10 minute until first log entry)
#-Djava.net.preferIPv4Stack=true
#-Djava.net.preferIPv6Addresses=false

# for Java debugging:
# -Xverbose:codegen

# do not modify below here #

# Check user
if [ `whoami` != $username ]
then
echo "ERROR: Only user $username is allowed to modify this service"
exit
fi

# Functions
check() {
pgrep -U $username -f $PROGRAM >/dev/null
}
kill() {
pkill -U $username -f $PROGRAM
}
stop() {
check
if [ $? -eq 0 ]; then
kill >/dev/null
# Make sure it is stopped before returning
until [ $? -ne 0 ]; do
sleep 1
check
done
check
if [ $? -eq 0 ]; then
echo "ERROR stopping application $PROGRAM"
exit 1
else
echo "Application $PROGRAM stopped"
#rm -f /var/lock/subsys/[init script name] <--- Perhaps, this lock file has to be removed, too.
fi
else
echo "Application $PROGRAM is not running"
fi
}
start() {
check
if [ $? -eq 0 ]; then
echo "Application $PROGRAM is already running"
exit 0
fi
export LC_ALL="de_DE@euro";
export LANG=german;
nohup java $JAVAPROGRAM 0<&- &>./logs/$PROGRAM-STDOUT.log &
echo "Application $PROGRAM started"
}
restart() {
stop
start
}
status() {
check
if [ $? -eq 0 ]; then
echo "Application $PROGRAM is running"
else
echo "Application $PROGRAM is stopped"
exit 1
fi
}
#Actions
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
status)
status
;;
*)
echo "Usage: $0 {start|stop|restart|status}"
exit 1
esac
exit 0

Mac Photos

2026-01-12T09:03:31Z

Torben: /* 2. Add missing locations */

[[Category:Software]][[Category:Apple]]
==MacOS Photos App==

===My Cleanup Workflow===
====1. Organize all photos into albums====
Using Smart Album to find missing:
Album is not Any
Album examples:
* 2024/240615 Some Event
* Documents
* Home
* Kid 1
* Kid 2
* Kids
* Me
* People
* Screenshots
* Sports/Jogging
* Sports/Cycling

====2. Add missing locations====
Using Smart Album to find missing:
Photo is not tagged with GPS
ways to set locations:
* manual Copy & Paste from other photo in Photo App via Menu -> Image -> Location (works also for multiple photos)
* manual set location text in Photo App via (I) -> Location, via copy paste of common locations from text/Excel file (works also for multiple photos)
* using Python osxphotos, see below
# pip install --upgrade osxphotos
osxphotos add-locations --selected --window 1H

====3. Add missing faces====
Using Smart Album to find missing:
Person is
(with empty input)

===Interact with Python===

====Copy missing locations from neighboring photos====
Install osxphotos
python3 -m pip install osxphotos
Update later
python3 -m pip install --upgrade osxphotos

Run in dry-run mode on selected photos
osxphotos add-locations --selected --window 1H

====Copy locations from GPX file====
Install
osxphotos install gpxpy
wget https://raw.githubusercontent.com/RhetTbull/add_photo_locations_from_gpx/refs/heads/main/add_photo_locations_from_gpx.py

Run in dry-run mode on selected photos
osxphotos run add_photo_locations_from_gpx.py --selected myFile.gpx
if time zone info is missing in photo, set UTC offset parameter like --offset +05:45:00

====Extract exif data e.g. gps location====
osxphotos query --selected --json | jq '.[].exif_info'
via Python and copy location to clipboard
import subprocess
import osxphotos

def copy_to_clipboard(text) -> None:
process = subprocess.Popen(["pbcopy"], stdin=subprocess.PIPE)
process.communicate(input=text.encode("utf-8"))

if __name__ == "__main__":
photosdb = osxphotos.PhotosDB()
try:
while 1:
results = photosdb.query(osxphotos.QueryOptions(selected=True))
for photo in results:
print(photo.original_filename, photo.date)
s = f"{photo.latitude},{photo.longitude}" # type: ignore
print(s)
copy_to_clipboard(s)
input("Press Enter for next image...")
except KeyboardInterrupt:
pass

====Links====
* https://www.reddit.com/r/ApplePhotos/comments/133vca9/geotag_photos_with_gpx_file/
* https://github.com/RhetTbull/osxphotos
* https://github.com/RhetTbull/add_photo_locations_from_gpx

Media Streaming Recording

2025-11-29T13:33:06Z

Torben:

Legal:
* Es ist in Deutschland grundsätzlich erlaubt, eine Privatkopie anzufertigen
* Die Allgemeinen Geschäftsbedingungen von Spotify etc. untersagen das aber
* [https://praxistipps.chip.de/musik-legal-bei-spotify-aufnehmen-alle-infos-im-ueberblick_135696 Quelle]

== Audio ==
e.g. Spotify, Deezer, Google Music, etc.

=== Mac Pre-Requirement ===
* install [https://github.com/ExistentialAudio/BlackHole BlackHole: macOS audio loopback driver]
brew install blackhole-2ch
* Reboot
* Set the new blackhole-2ch audio device as playback device

=== Recording ===
* install Audacity from https://www.audacityteam.org/download/mac/ (use the Version without Muse Hub, called "Universal dmg")
* set source to the blackhole-2ch audio device

=== Post-Recoring ===
* Split by Tracks
** Use Audacity [https://manual.audacityteam.org/man/label_sounds.html Analyze -> Label Sounds] feature
** Save to multiple files [https://support.audacityteam.org/audio-editing/splitting-a-recording-into-separate-tracks] via File -> Export Audio -> Export Range = Multiple Files
* Rename files via txt Track list
* Set ID3 Metadata

== Video ==
Chrome Browser Settings
-> System -> Disable Graphics Acceleration
prevents black screen recoding

Recording Software: [https://obsproject.com/ OBS Studio]

Lines of Code (cloc)

2025-11-02T06:28:59Z

Torben:

see https://github.com/AlDanial/cloc

Install on MacOS
brew install cloc
Run on git repo
cloc --vcs=git
Note, that it counts the package manager lock files.

Npm

2025-10-25T05:29:03Z

Torben: /* Globally Installed packages */

==pnpm==
# better and faster than npm
brew install pnpm
# or
npm install -g pnpm
# update single package
pnpm up vite
# update all dependencies
pnpm up

==npm Packagemanager for Javascript Packages==
===Globally Installed packages===
brew install node

npm install -g prettier eslint cspell
# list global packages
npm ls -g
# update all
npm update -g

===Install a package===
# install as main package
npm install --save vue
# install as dev package
npm install --save-dev cypress

===Update a package===
npm update cypress
force update
npm remove cypress
npm install --save-dev cypress

alternative: npm-check-updates
npm i -g npm-check-updates
ncu -u cypress
npm install
# or update all packages
ncu -u
npm install

===Windows Error about Execution_Policies===
npm.ps1 cannot be loaded because running scripts is disabled on this system. For more information, see about_Execution_Policies

open Admin PowerShell and run
Set-ExecutionPolicy RemoteSigned

Uberspace

2025-10-18T22:35:57Z

Torben: /* Python FastAPI */

[[Category:Webserver]][[Category:Linux]]
==Manuals==
https://manual.uberspace.de and https://lab.uberspace.de

==PHP==
see https://manual.uberspace.de/lang-php/

to restart
uberspace tools restart php

===php-fpm===
less /opt/uberspace/etc/$USER/php-fpm.conf

==Python FastAPI==
see https://lab.uberspace.de/guide_fastapi/

# var 1: venv
python3.11 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn
pip install fastapi.responses
pip install gunicorn uvloop httptools
deactivate

# var 2: not use venv
pip3.11 install --user fastapi uvicorn fastapi.responses gunicorn uvloop httptools

# config backend
uberspace web backend set /strava-be --http --port 9001
# --remove-prefix
uberspace web backend list

supervisorctl restart fastapi

~/fastapi/conf.py
#!/usr/bin/env python3.10

# apply changes via
# supervisorctl restart fastapi

import os

app_path = os.environ["HOME"] + "/fastapi"

# Gunicorn configuration
wsgi_app = "main:api"
bind = ":9001"
chdir = app_path
workers = 1
worker_class = "uvicorn.workers.UvicornWorker"

# errorlog = app_path + "/errors.log"
# accesslog = app_path + "/access.log"

~/fastapi/main.py
#!/usr/bin/env python3.10

# apply changes via
# supervisorctl restart fastapi

from fastapi import FastAPI
from fastapi.responses import JSONResponse
from pydantic import BaseModel

import subprocess

class StravaSession(BaseModel):
sessionId: str

api = FastAPI()

@api.post("/strava-be/activityStats2/")
async def activityStats2(session: StravaSession):
# return {session}
response = {}
# response = {"session": session.sessionId}
process = subprocess.run(
[
"python3.10",
"/var/www/virtual/entorb/html/strava/activityStats2.py",
session.sessionId,
],
capture_output=True,
)

if process.returncode == 0:
response["status"] = "ok"
response_code = 200
# response["process_stdout"] = process.stdout.decode()
else:
response["status"] = "error"
response_code = 400
# response["process_stdout"] = process.stdout.decode()
response["error_message"] = process.stderr.decode()
return JSONResponse(content=response, status_code=response_code)

# To test this app locally, uncomment:
# import uvicorn
# uvicorn.run(api, host="localhost", port=8001)

# curl -i -X POST "https://entorb.net/strava-be/activityStats2/" -H "Content-Type: application/json" -d '{"sessionId": "1234"}'

==Checkliste neuer U7 Account==
~/.bashrc und ~/.bash_profile von altem angeglichen
Logs aktiviert
uberspace web log access enable
uberspace web log apache_error enable
uberspace web log php_error enable
uberspace web errorpage 500 disable

==Probleme bei der Migration Uberspace U6 zu U7==
===0. E-Mails===
Es scheinen keine E-Mails mehr rein und raus zu gehen. (Account xxx@entorb.net).

Neuer Webmailer ist https://webmail.uberspace.de , dieser zeigt aber erst nach Abschluss der Migration auf das neue Postfach. Solange die Migration nicht abgeschlossen ist, muss im Mail Client (wie Thunderbird) dieser Login für den U7 Mail Account verwendet werden:
statt xxx@entorb.net -> xxx@entorb.uber.space

===1. HTTPS Zertifikat===
HTTPS Zertifikat war auf einmal nicht mehr gültig für https://www.entorb.net . Das hatte erst auch auf dem U7 funktioniert, nun plötzlich nicht mehr. UPDATE: Problem verschwand von alleine, hing vielleicht mit dem DNS Umzug zusammen.

===2. bestehende ezmlm Mailinglisten===
ezmlm ist gemäß [https://lab.uberspace.de/guide_ezmlm.html Anleitung] installiert, allerdings sagt mir
ezmlm-make -+ ~/ezmlm/oldlist ~/.qmail-oldlist oldlist entorb.uber.space
> ezmlm-make: fatal: unable to stat /etc/ezmlm/de: file does not exist

Wenn ich hingegen testweise einen neuen Verteiler anlege tut es:
ezmlm-make -A -u -m -5 xxx@entorb.net ~/ezmlm/mylist ~/.qmail-mylist
mylist entorb.uber.space
und diesen kann ich dann auch bearbeiten via
ezmlm-make -+ ~/ezmlm/mylist ~/.qmail-mylist mylist entorb.uber.space

Lösung:
in ezmlm/oldlist/ezmlmrc die entsprechende Zeile löschen

===3. Zugriff auf Dateien außerhab von ~/html===
Ich hatte einige Dateien (wie zB eine sqlite DB) in meinem Home (~/) liegen. Auf diese sollen Skripte die sich unter
~/html->/var/www/virtual/entorb/html
befinden zugreifen können. Das funktionierte nicht mehr, da SELinux hier rein grätscht

Beste Lösung: Die Dateien auf die der httpd zugreifen soll nach
/var/www/virtual/entorb/
verschieben und im Home Symlink hinterlassen.

Erster Ansatz: der SELinux Kontexte lässt sich so setzen (Siehe [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-security-enhanced_linux-working_with_selinux-selinux_contexts_labeling_files]
).
chcon -t httpd_sys_content_t file-name
half hier aber nicht:
ls -laZ /home/ |grep ento
drwx------. entorb entorb unconfined_u:object_r:user_home_dir_t:s0 entorb

===4. Perl Module, wie zB Excel::Writer::XLSX===
Installiert via
cpanm Excel::Writer::XLSX

Führe ich das Skript ~/html/test.pl in der Shell aus, wird das Modul
gefunden und geladen. Im Browser via https://entorb.net/test.pl wird
das Modul nicht gefunden.

Ursache 4.1 war dass SELinux den Zugriff des httpd auf /home/entorb/perl5 blockierte

Lösung:
~/perl5 Verzeichnis nach
/var/www/virtual/entorb/perl5
verschoben und im Home einen Symlink hinterlassen.

Dann zuerst lib::local installiert, dann andere Module.

Ursache 4.2 war dass ich in der .bashrc dies gesetzt habe:
eval $(perl -I$HOME/perl5/lib/perl5 -Mlocal::lib)
das musste ich im Perl Skript ebenfalls hinterlegen:
use lib ('/var/www/virtual/entorb/perl5/lib/perl5');
# aber !kein! use local::lib; !!!

===5. sendmail===
ich stellte fest, dass ich im U7 aus Python oder Perl CGI Skripten nicht mehr auf sendmail zugreifen kann. Über die Shell kommt die Mail sofort an. Beispiel:
8<---
import os
SENDMAIL = "/usr/sbin/sendmail"
to = "xxx@entorb.net"
subject = "testmail"
sender = "U7 <no-reply@entorb.net>"
body = "leer"
mail = f"To: {to}\nSubject: {subject}\nFrom: {sender}\nContent-Type:
text/plain; charset=\"utf-8\"\n\n{body}"
p = os.popen(f"{SENDMAIL} -t -i", "w")
p.write(mail)
p.close()
8<---
Lösung: statt Versand aus Web-Skripten/Seiten via sendmail, besser aus den Skripten ein Insert in eine SQLite DB machen und diese via cronjob und php mail() Funktion periodisch abarbeiten.

===6. Wiederholung der Migration===
Entweder einfach Migration auf dem U6 nochmal starten via
uberspace-move-account -u entorb
oder in Dashboard den Umzug abbrechen und neu starten -> anderer Server.

===7. MySQL DB Passwort angeblich zu kurz===
Das Migrationsskript hatte behauptet, dass mein MySQL Passwort zu kurz sei. Stimmte nicht, vermutlich kam es mit den Sonderzeichen durcheinander. Habe daher ein neues langes ohne Sonderzeichen vergeben.

==Streamlit==
see also [https://github.com/entorb/strava-streamlit/blob/main/README.md strava-streamlit readme]

Installation
pip3.11 install --user streamlit

Setup dir and config
mkdir ~/strava-streamlit
cd ~/strava-streamlit
mkdir .streamlit
vim .streamlit/config.toml

[browser]
gatherUsageStats = false

[server]
headless = true
port = 8501
baseUrlPath = "/strava-streamlit"

baseUrlPath is important when not running in web server document root

create minimal Streamlit app
vim src/app.py

import streamlit as st
st.title("Test")

start it manually as testrun (stop by ctrl+c)
streamlit run src/app.py

Add web backend
uberspace web backend set /strava-streamlit --http --port 8501

Access it via browser
https://entorb.net/strava-streamlit

Create Service
vim ~/etc/services.d/strava-streamlit.ini

[program:strava-streamlit]
directory=%(ENV_HOME)s/strava-streamlit
command=streamlit run src/app.py
# or
# command=python3.11 -O -m streamlit run src/app.py
loglevel=info

Start Service
supervisorctl reread
supervisorctl update
supervisorctl status
supervisorctl restart strava-streamlit

Check log
supervisorctl tail -f strava-streamlit
# as this is empty:
tail -f ~/logs/supervisord.log

Raspbian

2025-10-17T15:48:45Z

Torben: /* MQTT */

[[Category:Raspi]][[Category:Linux]]
Stuff for my Raspi / Raspi3 / Raspi4
===basics===

default login: pi/raspberry

initial package updates
sudo apt-get update
sudo apt-get dist-upgrade
sudo apt purge -y cloud-init
sudo apt-get autoremove --purge
sudo apt-get install vim mc git

===raspi-config===
sudo raspi-config
- Update
- Network : set hostname
- Interfacing options : Enable SSH
- Localization -> change Keyboard Layout
- advanced -> disable logging (to reduce SD usage)
- Locale -> en.US.UTF-8

for locale issues run
sudo dpkg-reconfigure locales

===SSH password-free-login===
ssh-copy-id pi@raspi

===Static IP===
(not used, set via DHCP server FritzBox instead)
sudo nmtui

===NTP Enable Time Sync===
sudo apt-get install systemd-timesyncd
sudo timedatectl status
# sudo timedatectl set-ntp true

===pi-hole, unbound, log in ram, ... ===
https://forum.kuketz-blog.de/viewtopic.php?f=53&t=8759
which IPv6 address?
ip a ls |grep fd00

===config.txt===
====Location====
Raspbian old:
sudo vim /boot/firmware/config.txt

Raspbian old:
sudo vim /boot/config.txt

Libreelec:
mount -o remount,rw /flash
nano /flash/config.txt
mount -o remount,ro /flas

====Power Saving====
see https://blues.io/blog/tips-tricks-optimizing-raspberry-pi-power/

====disable Wifi and Bluetooth====
[all]
# disable wifi
dtoverlay=disable-wifi
# disable bluetooth
dtoverlay=disable-bt

====Disable Onboard LEDs====
[pi4]
# Disable the PWR LED
dtparam=pwr_led_trigger=none
dtparam=pwr_led_activelow=off
# Disable the Activity LED
dtparam=act_led_trigger=none
dtparam=act_led_activelow=off
# Disable ethernet port LEDs
dtparam=eth_led0=4
dtparam=eth_led1=4

===cmdline.txt===
add "quiet" to remove kernel boot messages
sudo vim /boot/firmware/cmdline.txt
console=tty1 root=PARTUUID=eedf3243-02 rootfstype=ext4 fsck.repair=yes rootwait quiet cfg80211.ieee80211_regdom=DE

===CPU===
from https://raspberrypi.stackexchange.com/questions/103653/setting-the-maximum-clock-speed

set powersave at runtime
sudo apt install cpufrequtils
sudo cpufreq-set -g powersave

===systemctl_Services===
see [[Linux_Bash_Shell_Scripting#systemctl_Services]]

===Install uv===
https://docs.astral.sh/uv/getting-started/installation/
# install
curl -LsS f https://astral.sh/uv/install.sh | sh
# update
uv self update

===MQTT===
sudo apt install mosquitto mosquitto-clients

sudo vim /etc/mosquitto/mosquitto.conf
per_listener_settings true
allow_anonymous false
listener 1883
password_file /etc/mosquitto/passwd
# see http://www.steves-internet-guide.com/mqtt-username-password-example/

sudo vim /etc/environment
PW_MQTT=XXX

===Other===
grant access to tty to user pi
sudo usermod -a -G tty pi

Pi-hole

2025-10-17T12:39:13Z

Torben: /* Install */

[[Category:Raspi]][[Category:Software]]
===Install===
see https://docs.pi-hole.net/main/basic-install/
git clone --depth 1 https://github.com/pi-hole/pi-hole.git Pi-hole
cd "Pi-hole/automated install/"
sudo bash basic-install.sh
afterwards
pihole setpassword

===Config /etc/pihole/pihole-FTL.conf===
#; default = 1000/60
RATE_LIMIT=3000/60

#; I do not need long-time stats (default is 365 days!)
# MAXDBDAYS=2
MAXDBDAYS=0

#; frequency in minutes to update the DB
DBINTERVAL=60

#; DB location better keep in /etc/pihole/ where also the gravity.db is
#; DBFILE=/var/pihole/pihole-FTL.db

#; show all data
PRIVACYLEVEL=0

===Adlists===
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Spam/hosts
https://v.firebog.net/hosts/static/w3kbl.txt
https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt
https://someonewhocares.org/hosts/zero/hosts
https://raw.githubusercontent.com/VeleSila/yhosts/master/hosts
https://winhelp2002.mvps.org/hosts.txt
https://v.firebog.net/hosts/neohostsbasic.txt
https://raw.githubusercontent.com/RooneyMcNibNug/pihole-stuff/master/SNAFU.txt
https://paulgb.github.io/BarbBlock/blacklists/hosts-file.txt
https://adaway.org/hosts.txt
https://v.firebog.net/hosts/AdguardDNS.txt
https://v.firebog.net/hosts/Admiral.txt
https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
https://v.firebog.net/hosts/Easylist.txt
https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/UncheckyAds/hosts
https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
https://raw.githubusercontent.com/jdlingyu/ad-wars/master/hosts
https://v.firebog.net/hosts/Easyprivacy.txt
https://v.firebog.net/hosts/Prigent-Ads.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.2o7Net/hosts
https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt
https://hostfiles.frogeye.fr/multiparty-trackers-hosts.txt
https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt
https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt
https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareHosts.txt
https://osint.digitalside.it/Threat-Intel/lists/latestdomains.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
https://v.firebog.net/hosts/Prigent-Crypto.txt
https://bitbucket.org/ethanr/dns-blacklists/raw/8575c9f96e5b4a1308f2f12394abd86d0927a4a0/bad_lists/Mandiant_APT1_Report_Appendix_D.txt
https://phishing.army/download/phishing_army_blocklist_extended.txt
https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt
https://raw.githubusercontent.com/Spam404/lists/master/main-blacklist.txt
https://raw.githubusercontent.com/FadeMind/hosts.extras/master/add.Risk/hosts
https://urlhaus.abuse.ch/downloads/hostfile/
https://v.firebog.net/hosts/Prigent-Malware.txt
https://v.firebog.net/hosts/Shalla-mal.txt
https://zerodot1.gitlab.io/CoinBlockerLists/hosts_browser
https://raw.githubusercontent.com/mhhakim/pihole-blocklist/master/porn.txt
https://raw.githubusercontent.com/BlackJack8/iOSAdblockList/master/Regular%20Hosts.txt

InfluxDB

2025-10-17T11:48:06Z

Torben:

[[Category:Raspi]][[Category:Coding]][[Category:Python]]
==Install on Raspberry Pi in Raspbian==
===2025===
sudo apt install influxdb influxdb-client

Create DB and Users
# 1. create admin user
influx
create user uadmin with password 'password1' WITH ALL PRIVILEGES
exit

# 2. create DB and other users
export INFLUX_USERNAME=uadmin
export INFLUX_PASSWORD=password1

influx
USE raspi
create user uwrite with password 'password2'
create user uread with password 'password3'
grant read on raspi to uread
grant write on raspi to uwrite
exit

=== Old version===
Nice Howto: [https://www.circuits.dk/install-grafana-influxdb-raspberry/]

wget -qO- https://repos.influxdata.com/influxdb.key | sudo apt-key add -
source /etc/os-release
test $VERSION_ID = "7" && echo "deb https://repos.influxdata.com/debian wheezy stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
test $VERSION_ID = "8" && echo "deb https://repos.influxdata.com/debian jessie stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
test $VERSION_ID = "9" && echo "deb https://repos.influxdata.com/debian stretch stable" | sudo tee /etc/apt/sources.list.d/influxdb.list
sudo apt-get install influxdb
sudo service influxdb restart

==Setup==
===influxdb.conf===
sudo vim /etc/influxdb/influxdb.conf

[http]
# Determines whether HTTP endpoint is enabled.
enabled = true
# require login of user
auth-enabled = true
# prevent logging of each HTTP request to reduce load on ssd
log-enabled = false

[logging]
level = "warn"

in ~/.bashrc
export INFLUX_USERNAME=admin
export INFLUX_PASSWORD=password1

Runs per default on port 8086

==Administration==
===CLI: command line interface===
sudo apt-get install influxdb-client
influx -precision rfc3339 # for human readable time format
# or
influx
precision rfc3339

===User permissions===
user permissions are defined per database, so if you need only one level of permissions, one database might be enough
CREATE DATABASE raspi
USE raspi
suggestion: one user per permission
create user uadmin with password 'password1' WITH ALL PRIVILEGES
create user uwrite with password 'password2'
create user uread with password 'password3'
grant read on raspi to uread
grant write on raspi to uwrite

===Backup and Restore===
influxd backup -portable mypath
influxd restore -portable mypath

===Renaming a Database via creating a copy of its tables===
Attention: this will convert tags to fields is group by * missing
CREATE DATABASE mydb2
SELECT * INTO mydb2.autogen.table1 from mydb1.autogen.table1 group by *
DROP DATABASE mydb1

==General commands==

===SHOW commands===
SHOW DATABASES
SHOW MEASUREMENTS
SHOW RETENTION POLICIES
# display the field types of "table" myMeasurement
SHOW FIELD KEYS FROM myMeasurement
# show tags
SHOW TAG KEYS FROM myMeasurement
# for example to be used in Grafana variables of type Query
SHOW TAG VALUES FROM myMeasurement WITH KEY =~ //
# CQ
SHOW CONTINUOUS QUERIES

===SELECT statement===
precision rfc3339
# for human readable timeformat
USE myDB
SELECT * FROM myMeasurement WHERE time > now() - 3d
SELECT * FROM myMeasurement WHERE myField > 12.2

# cast field into different type into new measurement
SELECT watt::float INTO myMeasurement2 FROM myMeasurement1

# copy into new measurement:
SELECT watt::float, kWh_total_in::float, kWh_total_out::float INTO tasmota_MT681 FROM tmp
# use GROUP BY * to keep field types AND tags
SELECT watt_now, watt_last, kWh_total INTO tmp FROM Shelly GROUP BY *

Last value per day
SELECT last("kWh_total_in") as kWh_total_in, last("kWh_total_out") as kWh_total_out
FROM "tasmota_MT681"
WHERE 1=1
-- AND time > now() -2d
AND time < '2024-07-13T00:00:00+02:00'
GROUP BY time(1d) fill(previous)

===DELETE alternative===
DELETE is not supported in Influx V1

alternative:
# filter data into new tmp measurement and cast to float instead
# use GROUP BY * to keep field types and tags
SELECT watt::float, kWh_total_in, kWh_total_out INTO tmp FROM tasmota_MT681 WHERE kWh_total_in >0 GROUP BY *
# drop original measurement
DROP measurement tasmota_MT681
# restore from tmp
SELECT watt, kWh_total_in, kWh_total_out INTO tasmota_MT681 FROM tmp GROUP BY *
# cleanup
DROP MEASUREMENT tmp

==Retention Policies==
SHOW RETENTION POLICIES
# keep data for 10 years instead of 6 days (default)
CREATE RETENTION POLICY "years10" ON raspi DURATION 520w REPLICATION 1 DEFAULT
DROP RETENTION POLICY "years10" ON raspi

# infinite
CREATE RETENTION POLICY "inf" ON "raspi" DURATION INF REPLICATION 1

# if switching the default RETENTION POLICY old data will require a specific select
# altering the default RP:
ALTER RETENTION POLICY autogen ON raspi DURATION 90d SHARD DURATION 1d REPLICATION 1 DEFAULT
ALTER RETENTION POLICY autogen ON collectd DURATION 7d SHARD DURATION 1d REPLICATION 1 DEFAULT

==Continuous Query / Auto aggregation==
SHOW CONTINUOUS QUERIES

CREATE CONTINUOUS QUERY "cq_mt681_1d" ON "raspi" BEGIN \
INTO "inf"."tasmota_MT681_day_end" \
SELECT last("kWh_total_in") as kWh_total_in, last("kWh_total_out") as kWh_total_out \
FROM "tasmota_MT681" \
GROUP BY time(1d) fill(previous) \
END

DROP CONTINUOUS QUERY "cq_mt681_1d" ON "raspi"

==Connection via CURL==
curl -G http://raspi3:8086/query -u uwrite:password2 --data-urlencode "q=SHOW DATABASES"

===Insert via REST API and CURL===
# precision=s (second) is important for performance, if higher accuracy is needed use ms, not ns which is the default!)
VALUE="123.123"
curl -i \
-u uwrite:password2 \
-XPOST "http://localhost:8086/write?db=raspi&precision=s" \
--data-binary "myTable,myTag=SourceA myValue=$VALUE"

==SELECT, INSERT and UPDATE via Python==
===SELECT into Pandas DataFrame===
def read_data() -> pd.DataFrame:
"""Read data from Influx DB into DataFrame."""
client = connect2_df(credentials_read)
query = "SELECT * FROM {retention}.{measurement} WHERE ShellyNo = $ShellyNo"
result = client.query(
query.format(retention=retention, measurement=measurement),
bind_params={"ShellyNo": shelly_no},
)
df: pd.DataFrame = result[MEASUREMENT] # type: ignore
df.index.name = "time"
client.close()
return df

===INSERT / UPDATE via Pandas DataFrame===
def insert(df: pd.DataFrame) -> None:
"""Insert of df into Influx DB."""
client = connect2_df(credentials_write)
if len(df) > 0:
client.write_points(
df,
MEASUREMENT,
tag_columns=["room"],
protocol="line",
batch_size=1000,
retention_policy=RETENTION,
time_precision="s", # ns .. s
)
client.close()

===INSERT JSON data via InfluxDBClient===
see [https://www.influxdata.com/blog/getting-started-python-influxdb/]
from influxdb import InfluxDBClient
client = InfluxDBClient(host='192.168.178.31', port=8086, username='write', password='password2 ')
# client.create_database('raspi')
print(client.get_list_database())
client.switch_database('raspi')

json = [
{
"measurement": "brushEvents",
"tags": {
"user": "Carol",
"brushId": "6c89f539-71c6-490d-a28d-6c5d84c0ee2f"
},
"time": "2018-03-28T8:01:00Z",
"fields": {
"duration": 127
}
},
{
"measurement": "brushEvents",
"tags": {
"user": "Carol",
"brushId": "6c89f539-71c6-490d-a28d-6c5d84c0ee2f"
},
"time": "2018-03-29T8:04:00Z",
"fields": {
"duration": 132
}
}
]

if client.write_points(json, time_precision="s") != True:
print ("ERROR: Write to InfluxDB not successful")
else:
print ("data sent")

Grafana

2025-10-17T11:34:52Z

Torben:

[[Category:Raspi]][[Category:Software]]
===Install Grafana on Raspberry Pi in Raspbian===
[https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/ Grafana installation tutorial]

===Connect to InfluxDB===
[https://www.circuits.dk/install-grafana-influxdb-raspberry/ Howto combine Influx and Grafana (old)]

===Setup===
====grafana.ini====
in /etc/grafana/grafana.ini set the admin user and password
[dashboards]
# Number dashboard versions to keep (per dashboard). Default: 20, Minimum: 1
versions_to_keep = 5

====service====
# Start Grafana by running:
sudo service grafana-server start

Autostart at boot time
sudo systemctl enable grafana-server.service
sudo update-rc.d grafana-server defaults

default port: http://raspi:3000

===Migration from host to host===
# 1. adjust grafana.ini on target to match the one from source
sudo vim /etc/grafana/grafana.ini

# source
DIR=/var/lib/grafana
sudo scp -r $DIR user@target:incoming/

# target
DIR=/var/lib/grafana
sudo service grafana-server stop
sudo chown -R grafana:grafana incoming/grafana
sudo rm -rf $DIR
sudo mv incoming/grafana $DIR
sudo service grafana-server start

===Queries===
Display Table of Influx DB data
field "active" contains 1 if a given "hostname" is online
Query
WHERE time > now()-5m and field(active) != null
SELECT field(active) distinct() alias(online)
GROUP BY time($__interval), tag(hostname)
FORMAT AS Table
Min time interval = 5m
Visualization
Column Style -> Type Hidden for columns named online

===Convert cummulated kWh data to hourly or daily differences===
SELECT spread("kWh_total") FROM "Shelly3" WHERE ("room"::tag =~ /^$room$/) AND $timeFilter GROUP BY time($__interval) fill(null)
With "Query Options" -> "Min interval"=1h or 1d and maybe "Relative time" = now-7d

===Color of fill area based on threshold (e.g. red for negative values)===
Gradient mode -> Schema
Color scheme -> From threshold
Fill opacity -> 25

SSH

2025-10-17T10:52:00Z

Torben:

[[Category:Linux]]
see [https://help.github.com/articles/generating-ssh-keys] [https://en.wikipedia.org/wiki/Ssh-keygen]

===SSH password-free-login===
all in 1 go (exceute on ssh client machine)
ssh-copy-id user@remotehost

====On the ssh-client machine====
Use ssh-keygen to create private and public keys in the folder ~/.ssh/
ssh-keygen -t rsa
or
ssh-keygen -t rsa -C "email@host.com"
# ( or -t dsa )
passphrase can be empty
The public key is in the file ~/.ssh/id_rsa.pub

adding or changing passphrase
ssh-keygen -p

storing passphrase of client via ssh-agent (linux)
ssh-agent bash
ssh-add

====On the ssh server machine====
* append your public key to ~/.ssh/authorized_keys (you may have to create this file)
* set that files permissions to 0640
* set .ssh/ permissions to 0640

====Windows Client using Putty and PAgent====
* use PuTTYgen to generate a RSA key
** set password
** set comment, for example hostname of client machine
** store private and public keys
* edit generated public key
** remove linebreaks from key
** prepend ssh-rsa to key followed by a space. So it looks like ssh-rsa AAAAB3Nza...
* double click on private key to load it into Pageant (key manager)
* in Putty
** load your ssh connection
** put private key to Connection->SSH->Auth
** save connection
* on Server
** place public key without comments in ~/.ssh/authorized_keys, see above

===Mount using ssh===
apt-get install sshfs
now you can run
/usr/bin/sshfs -o idmap=user server:/home/USER /mount/DIR
The option idmap=user is important to map the servers and clients user id

(this does not work from fstab...)

===Fix for warning: setlocale: LC_CTYPE===
bash: warning: setlocale: LC_CTYPE: cannot change locale (UTF-8): No such file or directory
can be fixed on the SSH host by
sudo vim /etc/ssh/sshd_config
# remove this:
AcceptEnv LANG LC_*
# now restart ssh:
sudo systemctl restart ssh

Games for Kids

2025-09-21T11:50:28Z

Torben:

[[Category:Zocken]]
==Links==
===Mobile Devices===
* [https://www.stardewvalley.net Stardew Valley] [https://play.google.com/store/apps/details?id=com.chucklefish.stardewvalley @Google] [https://apps.apple.com/us/app/stardew-valley/id1406710800 @Apple]
* [https://pocketcitygame.com Pocket City 2] [https://play.google.com/store/apps/details?id=com.codebrewgames.pocketcity2 @Google] [https://apps.apple.com/us/app/pocket-city-2/id1533709428 @Apple]
* Minecraft [https://play.google.com/store/apps/details?id=com.mojang.minecraftpe @Google] [https://apps.apple.com/us/app/minecraft-play-with-friends/id479516143 @Apple]
* Angry Birds Classic [https://apps.apple.com/us/app/reds-first-flight/id1596736236 @Apple] (removed from Google)
* 2 Player Games [https://play.google.com/store/apps/details?id=com.JindoBlu.TwoPlayerGamesChallenge @Google] [https://apps.apple.com/app/2-player-games-the-challenge/id1465731199 @Apple]
* 1 2 3 4 Player Games [https://play.google.com/store/apps/details?id=com.JindoBlu.TwoPlayerGamesChallenge @Goolge] [https://apps.apple.com/app/1-2-3-4-player-games/id1635978552 @Apple]

===PC===
sorted recent to old
# [https://store.steampowered.com/app/1656930/Coridden/ Coridden]
# [https://store.steampowered.com/app/252110/Lovers_in_a_Dangerous_Spacetime/ Lovers in a Dangerous Spacetime]
# [https://www.stardewvalley.net Stardew Valley]
# Lego Games at Steam
## [https://store.steampowered.com/app/311770/LEGO_Pirates_of_the_Caribbean_The_Video_Game/ Pirates of the Caribbean]
## [https://store.steampowered.com/app/21130/LEGO_Harry_Potter_Years_14/ Harry Potter 1-4]
## [https://store.steampowered.com/app/204120/LEGO_Harry_Potter_Years_57/ Harry Potter 5-7]
## [https://store.steampowered.com/app/214510/LEGO_The_Lord_of_the_Rings/ Lord of the Rings]
## [https://store.steampowered.com/app/285160/LEGO_The_Hobbit/ Hobbit]
# [https://supertuxkart.net SuperTuxkart]

==Suggestions 2025 in German==
Hardware: PC mit 4 Xbox Controllern und Steam Games Store

Spiele: Fast ausschließlich kooperativ im SplitScreen als Familienevent. Kinder alleine auf Handy oder Tablett fast gar nicht.

===Kooperativ im SplitScreen===
Sortierung: für kleine Kinder oben

1 SuperTuxKart 
Open Source Mario Kart Clone, viel weniger überladen als das Original 
War unser erstes Spiel 
https://supertuxkart.net/

2 Stardew Valley 
unser all time Favorit, sehr schönes Indie Spiel 
(mit einem Tag-Nacht Rhythmus von 15 Minuten, mir das Ausschalten mit wenig Streit ermöglicht) 
Gibt es für alle Plattformen siehe 
https://www.stardewvalley.net/ 
(Glaube Mehrspieler geht nicht auf Habdy / Tablet)

3 Lego Spiele
Diverse Filme in Form von Lego Männchen. Fluch der Karibik, Harry Potter, Herr der Ringe. 
Nicht: Marvel und Ninjago, die waren mir zu sehr überladen 
Immer als 2-Spieler aufgesetzt, sprich ich muss leider zuschauen

4 Lovers in a Dangerous Spacetime 
Allein schon des Titels wegen 
mit bis zu 4 Spielern ein Raumschiff steuern und wie verrückt ballern um Hasen zu befreien 
https://store.steampowered.com/app/252110/Lovers_in_a_Dangerous_Spacetime/

5 Coridden 
Diablo Clone ohne Blut, dafür mit Gestaltwandlern für 4 Spieler. Erst vor einem Monat entdeckt und unser aktueller Favorit 
https://store.steampowered.com/app/1656930/Coridden/

===„Alleine Spiele“===
1 Pocket City 2 
Simcity für Kinder, Indie Spiel, schön gemacht 
https://pocketcitygame.com/

2 Stardew Valley 
(Siehe oben)

3 Minecraft

Docker

2025-07-18T09:43:06Z

Torben: /* Build you own image */

[[Category:Coding]][[Category:Linux]]
==Run container of public image==
docker run --name my_nginx -v ./html://usr/share/nginx/html -d -p 8080:80 nginx
# -v mounts/links ./html as volume
# -d (detach) runs container in background
# -p port mapping

==Build you own image==
Dockerfile = Building instruction for Docker image

# syntax=docker/dockerfile:1

FROM ubuntu:latest

# set timezone
ENV TZ=Europe/Berlin

# prevent keyboard input requests in apt install
ARG DEBIAN_FRONTEND=noninteractive

# install core packages
RUN apt-get update
RUN apt-get dist-upgrade -y
RUN apt-get install -y python3

Build Docker Image from Dockerfile:
docker build -t myImage .

To delete the image
docker rmi myImage

===Create Container from Image===
Create container and mount current working dir to /src
# Linux/MacOS:
docker run --name myContainer -it --mount type=bind,src="$(pwd)",target=/src myImage bash
# Windows:
docker run --name myContainer -it --mount type=bind,src="%cd%",target=/src myImage bash

Run Container
docker start -ai myContainer

To delete the container
docker rm myContainer

==Check Docker Resources==
sudo -i
# list of containers
docker ps
# top for containers
docker stats
# stats of single container
docker stats <id>
# disk space
docker system df
# Unused docker images
docker images --filter "dangling=true"

==Hacks==
===Windows Docker-Desktop: move image location===
from [https://www.kindacode.com/article/docker-desktop-change-images-containers-directory/?utm_content=cmp-true#The_Steps_For_Windows_Users_with_WSL_2_Backend]

first Exit Docker Desktop, than:
wsl --shutdown
wsl --export docker-desktop-data C:\tmp\docker-data.tar
wsl --unregister docker-desktop-data
wsl --import docker-desktop-data E:\docker\ C:\tmp\docker-data.tar --version 2

Git

2025-07-07T02:46:55Z

Torben: /* Modifiying History */

[[Category:Coding]]

==Git Basics==
===Basics===
====clone a repository====
git clone git@github.com:entorb/rememberthemilk.git

====perform a commit====
git add .
git commit -m "my commit message"
git push

===Git Settings===

==== !!! End of Line !!! ====
Force Git for Windows to use Linux end of line: LF (\n) instead of CRLF (\r\n)
git config --global core.autocrlf false
git config --global core.eol lf

To use git to fix bad line endings, add a .gitattributes file to the repo with this contents:
* text=auto eol=lf

====colorful output====
git config color.ui true

====log: one line per commit====
git config format.pretty oneline

===Branching===
# create new branch
git checkout -b history-change
# del branch
git checkout main
git branch -d history-change

====Rename branch master to main====
1. at github.com in the settings of the repo, rename the default branch name, for example at https://github.com/entorb/tools/settings/branches

2. update in local repo
git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

3. set default branch name in local git
git config --global init.defaultBranch main

==History==
===Reports===
====Git Log====
git log
git log --pretty=oneline --abbrev-commit

# list commits including the change summary, e.g. 1 file changed, 16 insertions(+), 2 deletions(-)
git log --date=short --pretty=format:"%cd %h %s" --shortstat

====Git Rev-list====
# Get the total commits by week
git rev-list --count HEAD --since=4.week
git rev-list --count main --since="Dec 1 2021" --before="Jan 3 2022"
====Report of merges====
var1
git --no-pager log --merges --pretty=format:'MyRepoName %cs %ae %s' | grep -v 'dependabot' >> ../git-merge-report.csv
var2
git --no-pager log --grep='#' --pretty=format:'MyRepoName %cs %ae %s' --after 2022-01-01 >> ../git-merge-report.csv
# note: in one repo the '--merges' flag did not find all merge PRs, so using --grep='#' instead

====Checking History====
Display all commits that modified a certain file
git log --no-decorate --pretty=format:"%h%x09%ad%x09%an%x09%s" --date=iso -- data/de-districts/de-district_timeseries-02000.tsv
# %x09 : tab

===Modifiying History===

====Merge 2 repos====
see https://github.com/entorb/tools/blob/main/git-tools/git-merge-2-repos.cmd

====delete a dir from historiy using git-filter-repo====
preferred as of 2022 history cleanup using [https://github.com/newren/git-filter-repo git-filter-repo]
pip install git-filter-repo
git clone xxx
cd xxx
# backup .git/config
cp .git/config ../config
git-filter-repo --prune-empty always --invert-paths --path draft --path old --path maps/out
# restore .git/config
cp ../config .git/config

git reflog expire --expire=now --all
git gc --prune=now --aggressive

git push -f

to remove certain files matching a glob from history
git-filter-repo --prune-empty always --invert-paths --path-glob "*.creds" --path-glob "data/*/*.json"

afterwards revert your local repo to the remote one
git fetch origin
git reset --hard origin/main
git pull

==== combine changes to README.md etc into single commit ====
see https://github.com/entorb/tools/blob/main/git-tools/git-hist-combine.sh

====delete a dir from historiy using bfg====
note: better use git-filter-rep (see above)

using [https://rtyley.github.io/bfg-repo-cleaner/ bfg]
https://github.com/entorb/tools/blob/main/git-tools/git-bfg-cleanup-delete-history-of-dir.cmd
git reflog expire --expire=now --all
git gc --prune=now --aggressive

====squash/merge old commits====
git rebase -i --root
# this opens a list of all commits in you editor
# replace pick by squash for the ones to

====delete complete commits history====
First Method from [https://gist.github.com/heiswayi/350e2afda8cece810c0f6116dadbe651]
# Check out to a temporary branch:
git checkout --orphan TEMP_BRANCH
# Add all the files:
git add -A
# Commit the changes:
git commit -am "Initial commit"
# Delete the old branch:
git branch -D main
# Rename the temporary branch to main:
git branch -m main
# Finally, force update to our repository:
git push -f origin main

Second Method via "deleting .git folder" also from [https://gist.github.com/heiswayi/350e2afda8cece810c0f6116dadbe651]
# Clone the project, e.g. `myproject` is my project repository:
git clone https://github/heiswayi/myproject.git
# Since all of the commits history are in the `.git` folder, we have to remove it:
cd myproject
# And delete the `.git` folder:
rm -rf .git
# Now, re-initialize the repository:
git init
git remote add origin https://github.com/heiswayi/myproject.git
git remote -v
# Add all the files and commit the changes:
git add --all
git commit -am "Initial commit"
# Force push update to the main branch of our project repository:
git push -f origin main

Alternative method: 
https://www.willandskill.se/en/deleting-your-git-commit-history-without-removing-repo-on-github-bitbucket/

==Hacks==
=== Merge GitHub Dependabot PRs ===
# switch to main branche and pull updates
current_branch=$(git branch --show-current)
if [ "$current_branch" != "main" ]; then
echo "branch was: $current_branch"
git checkout main
fi
git pull

# update origin
git fetch origin

# list all branches that start with "dependabot/" but not "springframework" and save to c:/tmp/dependabot-branches.txt
git branch -r | grep /dependabot/ | grep -v springframework >c:/tmp/dependabot-branches.txt
# create empty file
echo "" >c:/tmp/dependabot-changes.txt

while read -r branch; do
# remove "origin/" from the branch name
branch_name="${branch#origin/}"
echo "# $branch_name"
echo "# $branch_name" >>c:/tmp/dependabot-changes.txt
# switch to branch and pull latest version
git checkout "$branch_name"
git pull
# show changes to file pom.xml only
git diff main --unified=0 pom.xml >>c:/tmp/dependabot-changes.txt
echo ""
echo "" >>c:/tmp/dependabot-changes.txt
done <c:/tmp/dependabot-branches.txt

# back to main branch
git checkout main

# filter on changes only
grep -e '^[#\+\-] ' c:/tmp/dependabot-changes.txt >c:/tmp/dependabot-changes-filtered.txt

===Cleanup===
====Remove branch not (any more) on main====
prunes tracking branches not on the remote.
from https://stackoverflow.com/posts/28464339/timeline
git remote prune origin
list branches that have been merged into the current branch.
git branch --merged

===Reverting===

====reset to a commit====
get hash of commit
git log --oneline
git reset --hard HEAD~1
# git push -f

====revert a commit====
get hash of commit
git log --oneline
git revert <commit hash>
# git push -f

====Reset branch to remote branch====
from [https://stackoverflow.com/posts/1628334/timeline] Setting your branch to exactly match the remote branch can be done in two steps:
git fetch origin
git reset --hard origin/main
# git push -f

====Fixing/Overwriting authors/emails====
list authors
git shortlog -sn --all
git log --pretty --author="my@mail.com"
perform the history rewrite using global git config
(from https://stackoverflow.com/questions/750172/how-do-i-change-the-author-and-committer-name-email-for-multiple-commits/1320317#1320317)
git rebase -r --root --exec "git commit --amend --no-edit --reset-author"

===Squashing===
Squash last 3 commits
git reset --soft HEAD~3 &&
git commit

===revert local git to remote repository===
from [https://stackoverflow.com/posts/5361169/timeline]
git reset --hard HEAD
git clean -f -d
git pull

===GIT Commit to an existing Tag===
from https://gist.github.com/danielestevez/2044589
1) Create a branch with the tag
git branch {tagname}-branch {tagname}
git checkout {tagname}-branch

2) Include the fix manually if it's just a change ....
git add .
git ci -m "Fix included"
or cherry-pick the commit, whatever is easier
git cherry-pick {num_commit}

3) Delete and recreate the tag locally
git tag -d {tagname}
git tag {tagname}

4) Delete and recreate the tag remotely
git push origin :{tagname} // deletes original remote tag
git push origin {tagname} // creates new remote tag

5) Update local repository with the updated tag (suggestion by @wyattis)
git fetch --tags

This is based on https://gist.github.com/739288 thanks to nickfloyd for it

====Chmod====
set executable flag. see [https://stackoverflow.com/questions/21691202/how-to-create-file-execute-mode-permissions-in-git-on-windows]
git update-index --chmod=+x file

===Error handling===
Error:
git pull --tags origin main
[rejected] ... (would clobber existing tag)
Fixed by updating local tags with remote tags
git fetch --tags -f

===Tag handling===
Delete a tag locally
git tag -d myTag
Delete a tag remotely
git push --delete origin myTag

===Clean up via Garbage Collector===
git reflog expire --expire=now --all
git gc --prune=now --aggressive

==Scripts==
===Re-apply .gitignore===
see https://github.com/entorb/tools/blob/main/git-tools/git-gitignore-reapply.cmd
REM Remove everything from the git index in order to refresh your git repository:
git rm -r --cached .
REM Add everything back into the repo:
git add .
git commit -m ".gitignore re-applied"
git status
pause
git push

===pull on all repos===
for D in $(ls -d */); do
echo === $D ===
cd $D
# git status
current_branch=$(git branch --show-current)
if [ "$current_branch" != "master" ] && [ "$current_branch" != "main" ]; then
echo "branch was: $current_branch"
git checkout main
if [ $(git branch --show-current) != "main" ]; then
git checkout master
fi
fi
git pull --force
cd ..
done
read -p "Enter to close"

===revert on all repos===
for D in $(ls -d */); do
echo === $D ===
cd $D
current_branch=$(git branch --show-current)
if [ "$current_branch" != "main" ]; then
echo "branch was: $current_branch"
git checkout main
fi
git fetch origin
git reset --hard origin/main
git pull
cd ..
done
read -p "Enter to close"

==GitHub.com Tipps==
===create PR for another fork===
* on github.com propose an edit for 1 file
* this creates a branch in our repo based on the fork, called patch-1 or similar
* rename the branch at `github.com/<your name>/<your repo>/branches/yours`
* now you can locally pull from your main to fetch the new fork and locally add more modifications
* I suggest keeping "unmodified" base-branches of other forks, so easily create new PRs and branches based on them
* to update the fork from he fork's main, use github.com

===Copy commit from parent/upstream repo to forked repo===
ensure that the parent repo is listed as upstream:
git remote -v
git fetch upstream
# Create new branch
git checkout -b copy-commit-branch
git cherry-pick <commit-hash>
# Resolve conflicts (if any):
git cherry-pick --continue
git push origin copy-commit-branch
# Finally create a pull request

Short version
git fetch upstream
git cherry-pick <commit-hash>
git push

==Notes of 2020==
very nice tutorial: [https://rogerdudler.github.io/git-guide/index.de.html git - Der einfache Einstieg]

==Old notes==
[http://www.ralfebert.de/blog/tools/visual_git_tutorial_1/]
[http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html]

git init # create an empty git repository in the current folder
# set some settings
git config --global user.name "Torben Menke"
git config --global user.email "torben.menke@XXX.de"
git config --global color.ui auto
git config -l # shows configuration

git add somefile.txt # one file
git add somefolder # one folder
git add . # all

git commit
git commit -a -m "message" # commit all, message="message"

git pull ssh://tmenke@[IP]/[PATH] main
# edit
# commit
git push ssh://tmenke@[IP]/[PATH] main

git log
git log --pretty=oneline --abbrev-commit

Privacy

2025-07-06T05:48:08Z

Torben:

[[Category:Privacy]]
===Mobilfunkanbieter: Widerspruch gegen Erhebung von Bewegungsdaten===
https://www.kuketz-blog.de/empfehlungsecke/#widerspruch-bewegungsdaten

===Advertisement on Smartphone===
In order to get rid of personalized advertisement on your phone, *periodically* reset the phones ad id via 
Android: Settings -> Accounts & Sync -> Google -> Ads -> Reset advertising ID 
iPhone/iPad: Settings -> Privacy -> Advertising -> Reset advertising Identifier

===Aktualisierung 2018: Post verkaufte vor Bundestagswahl Daten von 34 Mio. Haushalten an CDU und FDP===
Nachdem die Post in bester Facebook-Manier vor der letzten Bundestagswahl sehr lukrativ aufbereitete aber anonymisierte Daten von 34 Mio. Haushalten an CDU und FDP verkloppt hat, habe ich mich mal schlau gemacht, wie man dem widersprechen kann. Ich versuchte es zunächst per [[Privacy#Widerspruchs_E-Mail_Vorlage|E-Mail]] und habe auch prompt eine automatische Antwort bekommen, die so klingt, als wenn ich nicht der erste bin der an die Adresse schreibt.

====Hintergrundinfos====
* [http://www.spiegel.de/politik/deutschland/deutsche-post-verkauft-daten-an-parteien-illegal-ist-das-nicht-a-1201057.html Spiegel Online]
* [https://www.zdf.de/comedy/heute-show/what-the-fakt-facebook-daten-skandal-deutsche-post-polizeigesetz-100.html Die Heute Show fasst es so zusammen:]
<blockquote>
Auch die [https://www.bild.de/bild-plus/geld/wirtschaft/deutsche-post/so-verhoekert-die-post-kundendaten-an-cdu-und-fdp-55260090.bild.html Deutsche Post verhökert Kundendaten], fand diese Woche die Bild am Sonntag heraus. Im Bundestagswahlkampf 2017 zahlten CDU und FDP für straßengenaue Analysen einer Post-Tochter: Die CDU baute darauf ihren Haustür-Wahlkampf auf, die FDP verschickte so Wahlwerbung an bestimmte Zielgruppen. Die Post-Tochter verfügt dabei über riesige Datenmengen. Die BAMS zitiert aus einem vertraulichen Papier der Post: „Für ca. 20 Mio. Häuser mit rund 34 Mio. Haushalten in Deutschland stehen mehr als 1 Milliarde Einzelinformationen zur Verfügung.“

Wie kommt das alles? Jeder, der in Deutschland eine Adresse hat, ist automatisch in den Post-Datenbanken. Um die Nutzung dieser Daten zu verhindern, müssen Verbraucher schriftlich widersprechen. Die Verbraucher können von der Deutschen Post Direkt GmbH auch Auskunft darüber fordern, welche Daten das Unternehmen über sie gesammelt hat und woher diese stammen. Dafür stellt die Verbraucherzentrale Schleswig-Holstein ein [https://www.verbraucherzentrale.sh/sites/default/files/2018-04/Musterschreiben%20VZSH%20Auskunft%20Datenspeicherung.pdf Musterschreiben] zur Verfügung. Auch die Datenschutzbehörde in NRW empfiehlt allen Verbrauchern, sich bei Post Direkt in Troisdorf (z.B. an datenschutz@postdirekt.de) zu erkundigen, was über sie gespeichert wurde.
</blockquote>

====Widerspruchs E-Mail Vorlage====
Ich habe diese [https://www.datenschutz-berlin.de/pdf/adressenhandel/BlnBDI_Deutsche_Post_Direkt_GmbH_Widerspruch.pdf Vorlage] verwendet, angepasst und per [mailto:Datenschutz@postdirekt.de E-Mail] verschickt. Alternativ kann man auch diese [https://www.verbraucherzentrale.sh/sites/default/files/2018-04/Musterschreiben%20VZSH%20Auskunft%20Datenspeicherung.pdf Vorlage aus SH] verwenden oder irgendeine andere.

Sehr geehrte Damen und Herren,

hiermit widerspreche ich der Verwendung meiner personenbezogenen Daten durch die Deutsche Post Direkt GmbH für Zwecke der Werbung sowie der Markt- und Meinungsforschung gem. § 28 Abs. 4 Bundesdatenschutzgesetz (BDSG) bzw. Art. 21 Abs. 2 EU-Datenschutz-Grundverordnung (DSGVO). Zudem bitte ich Sie, meine Anschrift in Ihre hausinterne Sperrliste aufzunehmen, um eine zukünftige Nutzung oder Übermittlung für Zwecke der Werbung und der Markt- und Meinungsforschung zu vermeiden.

Darüber hinaus fordere ich Sie nach § 34 BDSG bzw. Art. 15 DSGVO auf, mir unentgeltlich Auskunft zu erteilen über die bei Ihnen über mich gespeicherten Daten, den Zweck der Speicherung, die Herkunft der Daten, sowie alle Personen und Stellen, gegenüber denen meine Daten offengelegt wurden.

Ihre Bestätigung meines Widerspruchs sowie die oben erbetene Auskunft erwarte ich bis spätestens ''Datum (habe 30 Tage Zeit gegeben)'' an unten genannte Adresse.

Mit freundlichen Grüßen 
''Name'' 
wohnhaft in ''Adresse1'' 
ehemals in ''Adresse2'' 
ehemals in ''Adresse3''

===Aktualisierung 2014===
Updated Software is here: [[Links#Privacy]] and [[Links#Firefox_Add-ons]]

===Aktualisierung 2014: Unerwünschte Werbung, Post Direkt, Robinson Liste===
Nachdem ich Anfang 2013 ungefragt Werbung von [http://www.kabeldeutschland.de Kabel Deutschland] bekam, wunderte ich mich woher diese Firma meine Adresse hatte. Im Kleingedruckten stand als Quelle [http://www.schober.de Schober Information Group]. Schober seinerseits hat einen [http://www.presseportal.de/pm/58743/2305574/deal-fuer-die-zukunft-deutsche-post-direkt-und-schober-information-group-kooperieren-bei-consumer Deal] mit der [http://portal.postdirekt.de/ Post Direkt] (einer Tochter der DE Post) geschlossen, die wiederum damit wirbt, dass sie Daten von 37 Mio Haushalten anbietet. (siehe auch diesen [http://www.golem.de/1201/89380.html Golem Artikel])

Nachdem ich allen 3 Firmen über die jeweiligen Kontaktformulare auf den Internetseiten mitteilte, dass ich einer Nutzung und Weitergabe widerspreche und erfahren möchte was mit meinen Daten bisher passiert ist, bekam ich überraschender Weise schnell 3 nette Briefe mit Versprechen mich auf die jeweiligen in Zukunft nicht mehr anzuschreiben. Am interessantesten war der von Schober, denn man teilte mir mit, dass man meine Daten nicht von der Post, sondern vom [http://www.otto.de Otto Versand] erworben wurden und dass meine Daten an die '''[http://www.rundfunkbeitrag.de/ GEZ]''' verkauft worden sind.

Es wurde mir ein Eintrag in die [http://de.wikipedia.org/wiki/Robinsonliste Robinsonliste] nahegelegt, welchen ich hiermit weiterempfehlen möchte: http://www.ichhabediewahl.de

===Aktualisierung 2013===
Lektüre von [https://www.awxcnx.de/download/privacy-handbuch.pdf] und diese Seite [https://www.anonym-surfen.de] sind empfehlenswert.

===Aktualisierung 2012===
Sehr schöne Übersicht + Lösungen aus [https://www.datenschutzzentrum.de/tracking/schutz-vor-tracking.html Schleswig Holstein]. Auch nett die aus [http://www.datenschutz-hamburg.de Hamburg].

====Tracking Sites====
Tracker sind kleine Schnippsel, die den Webseitenbetreibern Zugriffsstatistiken bringen, aber den Schnippselanbietern (z.B. Google Analytics, Facebook via Like Buttons außerhalb von Facebook) verraten auf welchen verschieden Seiten ich so surfe. Aussage von [http://www.spiegel.de/extra/0,1518,748826,00.html Spiegel Online]: selbst wenn du grad nicht bei Facebook eingeloggt bist, bekommen die doch deine IP Adresse und wissen damit wo du so rumgesurft bist. Ziemlich doof finde ich...

Hier mehr dazu [https://www.datenschutzzentrum.de/tracking/schutz-vor-tracking.html], dort wird das Browser Addon "Ghostery" empfohlen, das mir sehr gut gefällt.

[http://www.ghostery.com/download Ghostery] ist ein großartiges Browser-Addon das Tracking Skripte blockiert.

Einstellungen (zu erreichen über das Geist-Symbol) die man meiner Meinung nach verwenden sollte:
Im Firefox heißen die Optionen so: (in anderen Browsern anders)
* "GhostRank aktivieren" EIN (Geschmackssache)
* "Automatische Aktualisierung der Zählpixelliste aktivieren"
* Blockeroptionen -> Zählpixel -> Select all !!!
* Blockeroptionen -> Cookies hab ich nicht aktiviert
* Advanced (oben) -> Allgemeine Optionen -> "Warnmeldung einblenden" AUS (ist Geschmackssache)
* Advanced -> Auto Update -> "Block new elements by default" EIN
* Advanced -> Leistungsoptionen -> "Flash- und Silverlight-Cookies beim Verlassen löschen" EIN

====Facebook Like Buttons====
Lösung:
Ich verwende 2 Browser, z.B.
- InternetExplorer oder Opera nur für FB
- Firefox für alles außer FB
Dazu dass Firefox Addon: <del>'''BlockSite'''</del> [https://addons.mozilla.org/de/firefox/addon/ublock-origin/ uBlock Origin]
mit den Filter Einträgen:
facebook.*
fbcdn.*
gern auch noch was gegen die TrackingSeite
ivwbox.*

====Manually====
Google Analytics
Wenn man auch gleich noch die andere Datenkrake Google etwas beschneiden will, sollte man Google Analytics (Webserverstatistic) blockieren. Dazu gibt es ein Browser Plugin von [http://tools.google.com/dlpage/gaoptout Google] ;-):

====Zu Cookies====
Ich halte es für eine gute Idee im Firefox einzustellen, das Cookies beim Schließen des Browsers gelöscht werden. Dazu Einstellungen -> Datenschutz

===Bit Outdated Version of this Text===

I am blocking the following pages using my browsers blocking tool (Opera: Block Content, or Firefox: Plugin BlockSite) If I want to access Facebook I now use a different browser, that is used only for this kind of pages and which has no blocking set up.

*.facebook.* (because of like button)
*.fbcdn.* (belongs to Facebook)
*.studivz.*
*.ivwbox.* (statistics)
*.twitter.* (retweet, etc)
*.google-analytics.*
*.googlesyndication.*
*.googleadservices.*
*.urchin.*
*.adservices.google.com*
*.video-stats.video.google.com*
*.apps5.oingo.com* (Microsoft.Typo-Patrol)
*.appliedsemantics.*

See also [[How_to_get_rid_of_Google_Analytics]] for blocking Google Analytics completely and system wide via hosts file.

Local private LLM

2025-06-01T14:40:41Z

Torben: /* Installation */

== Motivation ==
* Large language models (LLMs) are very good at analyzing (large) texts
* There are multiple free-to-test providers in the internet
* Sensitive private or confidential data (like diary/journal, medical records, etc.) should '''NOT''' be shared with any external service

== Solution ==
Run your own LLM

== Installation ==
I use the nice open-source tool [https://ollama.com Ollama] to manage and host the LLM.

== Running ==
Start in terminal/command window by passing a model (the llama3.2 was not working well for me, see below)
ollama run llama3.2:latest

A list of supported LLM models can be found [https://ollama.com/search here].

For me the [https://ollama.com/library/qwen3 qwen3] model worked best, so far.

Besides the model, you need to select a model size, that fits to your hardware.

For me:
MacBook Pro M1 16GB
-> models up to 8b work, but with patience
Windows PC with gaming graphics card (GPU) GeForce RTX 3060 12GB
-> models up to 14b run very smooth

To check if the model fits into you GPU memory, first run
ollama run <model>
than open a second terminal window and run
ollama ps
to see where the model is stored.

== Usage ==
=== chat commands ===
/clear : clear current session
/bye : exit

=== command line commands ===
show currently running model(s) and where it is stored (RAM or GPU RAM)
ollama ps

stop model (automatically done after 5min inactivity)
ollama stop llama3.2:latest

delete model
ollama rm llama3.2:latest

===Remote access===
Set env variable
OLLAMA_HOST=192.168.0.123:11434

===Local storage of the models ===
[https://github.com/ollama/ollama/blob/main/docs/faq.md#where-are-models-stored where-are-models-stored]
macOS: ~/.ollama/models
Linux: /usr/share/ollama/.ollama/models
Windows: C:\Users\%username%\.ollama\models
change via env var OLLAMA_MODELS (ollama service restart needed)

=== Links ===
* [https://ollama.com/library/deepseek-r1 Deepseek Models]
* [https://github.com/ollama/ollama/blob/main/docs/faq.md FAQ]

Links

2025-05-24T05:07:00Z

Torben: /* Security */

[[Category:Software]]
==Online==

===Websites I built===
* [https://www.mtv-pattensen.de MTV Pattensen]
* [http://www.fliesen-wochnik.de Fliesen-Wochnik]
* [https://www.huenert-kramp.de/joomla/ Hünert & Kramp]

===Blogs===
* [http://blog.fefe.de Fefe ]
* [http://www.bildblog.de BildBlog ]

===Translators===
* [https://www.deepl.com Translator: Deepl]
* [https://dict.leo.org Translator: German <-> English (Leo)]
* [https://en.pons.com/translate Translator: German <-> English (Pons)]
* [https://www.dict.cc Translator: German <-> English (DictCC)]
* [http://deutsch-schwedisches-woerterbuch.elch.nu/lexikon.php Translator: German <-> Swedish]
* [http://lexin2.nada.kth.se/swe-eng.html Translator: Swedish <-> English]
* [https://www.deutsch-plattdeutsch.de Translator: German <-> Low German (Sprüche)]

===Weather Forecast===
* [https://www.lightningmaps.org/realtime LightningMaps]
* [https://wetter.rtl.de Wetter.de]
* [https://wetterstationen.meteomedia.de/messnetz/forecast/102470.html Meteomedia ]
* [https://www.windguru.cz Windguru Surf Weather Forecast]
* [https://muchoviento.net Muchoviento Wind Forecast]
* [https://www.windfinder.com Windfinder]

===Nice Webpages===
* [https://ihatemoney.org/ I hate Money: Split Bills / Expenses]
* [https://www.xe.com/ucc/ Currency Converter ]
* [https://filext.com File-Extension-Glossar]
* [https://www.tomshardware.com/charts/ Tom's Hardwareguide - Charts ]
* [https://www.librarything.com Library Thing ]
* [https://www.vorleser.net Vorleser - Free Audiobook ]
* [https://www.gutenberg.org Project Gutenberg - Free eBooks ]
* [https://www.dafont.com DaFont - Free Font Archiv]
* [https://www.gapminder.org Gapminder - very pretty custom statistics on the world]

===Cool Online-Tools===
* [https://www.random.org/passwords/ Random Password Generator]
* [https://tinyurl.com TinyURL]
* [https://www.webyield.net/link_checker.html Link-Checker for websites]
* [https://www.fliptext.net FlipText ]
* [https://dudle.inf.tu-dresden.de dudle: alternative to Doodle]
* [https://www.sibiller.de/anagramme Anagramm Generator]
* [https://www.techpowerup.com/review/ TechPowerUp - PC Hardware Performance Reviews]
* [https://downr.org Download YouTube: Video or Audio]

===Sport===
* [https://gotoes.org/strava/Combine_GPX_TCX_FIT_Files.php Merge GPX/FIT files for upload to Strava etc]
* [https://trackprofiler.com Trackprofiler edit GPX(GPS) files] (alternative to gpsies.com
* [https://cultureplot.com/strava-heatmap/ Heartmap for GPX files (and Strava API)] and [https://github.com/OGladfelter/strava_heatmap GitHub Repo]

===Privacy===
* [https://haveibeenpwned.com have i been pwned - Check if your email address is in a data breach]
* [https://www.torproject.org Tor Privacy Browser Bundle, based on Firefox]
* [https://www.eff.org/https-everywhere Firefox Plugin https-everywhere]
* [https://disconnect.me Firefox Plugin disconnect.me blocks trackers]
* [https://www.startpage.com StartPage WebSearch] or [https://ixquick.com Ixquick] as alternative to Google
* [https://palava.tv secure browser based video chat]
* [https://anonbox.net CCC Anonbox 1 day eMail] or [http://www.mailinator.com] or [http://www.temporaryinbox.com] or [http://www.10minutemail.com]
* [https://frank-geht-ran.de Wegwerftelefonnummer 01631737743]
* [https://bugmenot.com bugmenot: Share Logins]
* [https://prism-break.org Prism Break]

===Security===
* [https://www.grc.com/x/ne.dll?bh0bkyd2 Shields Up: Router Test]
* [https://www.ssllabs.com/ssltest/analyze.html SSL Labs HTTPS Check]

==Software I like==
===Software I like (Windows, Linux and Mac)===
* [https://dbeaver.io DBeaver Multi-DB-Admin tool]

===Software I like (Linux only)===
* [https://www.krusader.org Krusader]
* [https://kdirstat.sourceforge.net/ KDirStat - visualise disk usage]
* [https://www.vysor.io Vysor - Mirror Android Screen to PC (Chrome)]
* [https://goaccess.io GoAccess - real-time web log analyzer]

===Software I like (Windows only)===
====Making Windows usable/productive====
* [https://www.ghisler.com Total Commander]
* [https://notepad-plus.sourceforge.net Notepad ++] here some [[Notepad++|tipps]]
* [https://www.outertech.com/en/clipboard-manager Clipboard Manager with History]
* [https://stefansundin.github.io/altdrag/ AltDrag - Move Windows like in Linux]
* [https://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty SSH client]
* [https://winscp.net WinSCP sftp for windows]
* [https://getgreenshot.org/ Greenshot Screenshot Tool]
* [https://sourceforge.net/projects/unxutils/ UnixUtils] - Port of Unix/Linux tools like wget, grep,... for windows, see [[Windows_Batch_Scripting#UnixUtils|tipps]]

====Directory Size Analysis====
* [https://diskanalyzer.com/download WizTree]
* [https://windirstat.info WinDirStat - visualise disk usage]
* [https://www.jgoodies.com/freeware/jdiskreport/ JDiskReport - visualise disk usage and files per folder]

====Repairing / fixing / tweaking Windows====
* [https://gallery.technet.microsoft.com/scriptcenter/Reset-Windows-Update-Agent-d824badc Microsoft Repair Tool for Windows Updater]
* [https://www.ccleaner.com CCleaner - cleaning temp, cache, etc]
* [https://xp-antispy.org XP-Antispy]
* [https://www.chip.de/downloads/O-O-ShutUp10_82318496.html O&O ShutUp10] privacy tuning for Windows 10

====Photo Tools====
* [https://gimp-win.sourceforge.net/ Gimp (Windows-Installer)]
* [https://inkscape.org Inkscape] Here some [[Inkscape| tipps]]
* [https://www.foto-freeware.de/resize.php Picture-Batch-Resize]
* [https://adionsoft.net/fastimageresize/ adionSoft Fast Image Resizer ]
* [https://www.mediachance.com/hdri/index.html Dynamic Photo HDR]
* [https://www.darkleo.com DarkShot 2: Simple Screenshot Tool ]
* [https://www.faststone.org FastStone Image Viewer ]
* [https://www.andreaplanet.com/andreamosaic/ Mosaic Generator ]
* [https://www.sixdots.de/mosaik/de/ Mosaic Generator #2]

====Audio====
* [https://www.wma-mp3.org Free WMA to MP3 Converter]
* [https://www.mp3mymp3.com MP3 my MP3 records any sound from your PC, splits on silence]
* [https://xrecode.com xrecode II] CD Ripper and Sound Cross Encoder

====Video====
* [https://www.formatoz.com FormatFactory - Video Converter / Encoder] see [[Video shrinking]]
* [https://www.kinovea.org Kinovea Sport Video Analyse Tool]

====Other====
* [https://www.libreoffice.org LibreOffice.org] here some [[LibreOffice / OpenOffice|tipps]]
* [https://www.accesspdf.com/pdftk/ pdf Toolkit]
* [https://www.gimp.org Gimp]
* [https://www.vim.org gVIM]
* [https://hamachi.en.softonic.com/ Hamachi - Virtual Private Network (VPN)]
* [https://www.der-hammer.info/terminal/ HTerm]
* [https://www.workrave.org Workrave: forces you to take breaks]
* [https://sourceforge.net/projects/synergy2/ Synergy: 2 PCs, one Mouse and Keyboard]
* [https://free.grisoft.com/ AVG Anti-Virus (free)]
* [https://www.free-av.de/ AntiVir Anti-Virus (free)]
* [https://www.rejetto.com/hfs/ HFS - Http File Server (easy sharing of files)]
* [https://www.castelware.de/downloads_freeware.html CWSysinfo: Displays a lot of system-info (even your Windows serial no.)]
* [https://www.magicaljellybean.com/keyfinder/ Magical Jelly Bean Keyfinder: Displays Serial Numbers]
* [https://freepdfxp.de/ FreePDF XP for creating PDF Documents via a printer-driver]
* [https://www.deepburner.com DeepBurner (free CD/DVD Burning Tool)]
* [https://www.2brightsparks.com/freeware/freeware-hub.html SyncBack Backup-Tool]
* [https://www.winsplit-revolution.com WinSplit - nice tool for arranging windows ]
* Fill empty Space with zeors for VM compression: [https://docs.microsoft.com/de-de/sysinternals/downloads/sdelete MS sdelete] or nullfile [http://www.feyrer.de/g4u/nullfile-1.02.exe 32bit] [http://www.feyrer.de/g4u/nullfile-1.01_64bit.exe 64bit]
* [https://www.keyfocus.net/kfws/ KeyFocus WebServer]
* [https://byshynet.com/software.php?id=3 MouseMover]
* [https://www.chip.de/downloads/ArtMoney-Special-Edition_32716630.html ArtMoney: Game Cheating Tool]

===Firefox Add-ons===
see [[Firefox#Firefox Add-Ons|Firefox -> Firefox Add-Ons]]

==Games==
===Free Games===
see https://www.golem.de/news/gratisspiele-wirklich-kostenlose-games-im-all-und-in-der-hoelle-2401-181559.html

====Adventures====
* [https://zak2.org Zak McKracken 2: Adventure]
* [https://www.chip.de/downloads/Maniac-Mansion-Deluxe_13013383.html Maniac Mansion Deluxe]

====Racing====
* [https://supertuxkart.net Super Tux Kart Racing (Super Mario Clone)]
* [https://www.chip.de/downloads/TrackMania-Nations-Forever_31482232.html TrackMania Nations Forever]
* [https://phoboslab.org/wipegame/ Wipeout Browser Version]

====Shooter====
* [https://www.soldat.pl Soldat: 2D Shooter]
* [https://tremulous.net Tremulous: 3D Shooter using Quake 3 Engine]
* [https://heroesinthesky.gamigo.de Heroes in the Sky]
====Other====
* [https://www.openra.net Open RA (C&C Red Alert Clone)]
* [https://www.runesofmagic.com Runes of Magic (WoW-Clone)]
====Browser Games====
* [https://www.decisionproblem.com/paperclips/index2.html Universal Paperclips]
* [https://adarkroom.doublespeakgames.com/ A Dark Room]
* [https://orteil.dashnet.org/cookieclicker/ Cockie Clicker]
* [https://candybox2.github.io/ Candy Box 2]
* [https://neal.fun/password-game/ The Password Game]
* [https://gandalf.lakera.ai make Gandalf reveal the secret password]

PostgreSQL

2025-05-21T06:56:25Z

Torben: /* Find and Delete Locks / Processes */

[[Category:Coding]]
==Connect via single line connection string==
postgresql:user:password@host:5432/database

==SELECT==
===Date handling===
====Filter on date====
SELECT * FROM table WHERE created > '2020-06-22 00:00:00+02';
SELECT * FROM table WHERE created > current_timestamp - '3 hours'::interval;
SELECT * FROM table WHERE created > current_date - 91; -- days
SELECT * FROM table WHERE created > now() at time zone 'utc' - interval '10 minute';
# day of week filter
SELECT * FROM table WHERE EXTRACT(DOW FROM created) = 0;

====date_trunc: convert date====
SELECT date_trunc('hour', t.created) as hour FROM table t;
SELECT date_trunc('day', t.created)::date AS created_date FROM table t;
SELECT date_trunc('week', t.created)::date AS week FROM table t;
SELECT date_trunc('month', t.created)::date AS month FROM table t;

====extract====
SELECT EXTRACT (hour from datecol) FROM myTable
SELECT EXTRACT (isodow from datecol) FROM myTable -- weekno
SELECT EXTRACT (dow from datecol) FROM myTable -- weekday
SELECT EXTRACT(epoch FROM closed - created) / 3600 AS hours FROM myTable -- difference in hours

====other====
# gen list of dates
SELECT i::date AS day FROM generate_series('2020-01-01', CURRENT_DATE, '1 day'::interval) i

====Group by date and add missing days with zero count====
WITH all_days AS (
SELECT generate_series('01.01.2023', current_date, '1 day') :: date AS date
), count_per_day AS (
SELECT
created :: date AS date,
COUNT(*) AS count
FROM my_table
GROUP BY 1
)
SELECT
all_days.date,
COALESCE (count, 0) AS count
FROM all_days
LEFT OUTER JOIN count_per_day ON all_days.date = count_per_day.date
ORDER BY
1 DESC;

===cast types===
select t.id, 'matched filter1'::character varying(16) from table t

-- 1. convert timerange to seconds
-- 2. perform average
-- 3. cast to nummeric to allow for rounding
SELECT round(CAST(EXTRACT(EPOCH FROM avg (s.closure_date - s.date_of_start)) /3600 as numeric),1) AS avg_hours_total_avg from myTable s group by cloumn1

-- convert date range to days
SELECT EXTRACT (DAY FROM (current_date - date_due)) AS days FROM myTable

====leading zeros for integer values====
select TO_CHAR(plant_id, 'fm0000') from myTable

===rounding of REALS===
round(lower_specification_limit::numeric,6) as lower_specification_limit,

===EXCEPT (=MINUS in Oracle)===
from [http://www.postgresqltutorial.com/postgresql-tutorial/postgresql-except/]
SELECT column_list
FROM A
WHERE condition_a
EXCEPT
SELECT column_list
FROM B
WHERE condition_b;

===RowSum / LAG===
WITH daily_sum AS (
SELECT created, sum(count_finished)
FROM c_report_timeseries
WHERE department_id > 0
GROUP BY 1
ORDER BY 1
)
SELECT created, sum, sum - lag(sum) over (order by created) as increase
FROM daily_sum;

===Grouping and stats: min, max, avg, median===
select d.name "Department",
count(*) ,
min(i.modified-i.created),
max(i.modified-i.created),
avg(i.modified-i.created),
PERCENTILE_CONT(0.5) WITHIN GROUP(ORDER BY (i.modified-i.created) ) as median
EXTRACT(EPOCH FROM PERCENTILE_CONT(0.5) WITHIN GROUP(ORDER BY (i.modified-i.created)))::int AS median_seconds
FROM table i

===Substring===
-- 087249731000011 -> 087249731-0000-11
SELECT id, plant_id, material_number
, substring (material_number FROM 1 for 9) || '-' || substring (material_number FROM 10 for 4) || '-' || substring (material_number FROM 14 for 2) AS "MatNoNew"
FROM d_template t
WHERE 1=1
AND t.department_id = 50012487
AND material_number NOT LIKE '%-%'
AND length(material_number) = 15
;

===sequence.nextval===
SELECT nextval('mysequence');
===Pivot / longtable to shorttable===
WITH data AS (
SELECT
date_trunc('week', created)::date AS "Date",
myType,
count (*)
FROM myTable
GROUP BY 1,2
)
SELECT "Date",
max("count") FILTER (WHERE myType= 'D') AS "cntD",
max("count") FILTER (WHERE myType= 'S') AS "cntS
max("count") FILTER (WHERE myType= 'M') AS "cntM"
FROM data
GROUP BY 1
ORDER BY 1 ASC

===Weighted Average===
WITH my_table AS (
SELECT id, property, val1, val2, val3
CASE
WHEN created>current_date - 365 THEN 3
WHEN created>current_date - 730 THEN 2
ELSE 1
END AS weighting_factor
)
SELECT
sum(val1*weighting_factor)/sum(weighting_factor) AS "val1w",
sum(val2*weighting_factor)/sum(weighting_factor) AS "val2w",
sum(val3*weighting_factor)/sum(weighting_factor) AS "val3w"
FROM my_table
GROUP BY property

==INSERT==
===Insert or Update if conflict===
INSERT INTO login_cnt (user_id)
VALUES (%s)
ON CONFLICT (user_id, "date")
DO UPDATE SET count = login_cnt.count + 1;

-- login counter table
CREATE TABLE
login_cnt (
user_id int2 NOT NULL,
"date" date DEFAULT current_date NOT NULL,
count int2 DEFAULT 1 NOT NULL,
CONSTRAINT pkey_login_cnt PRIMARY KEY (user_id, date)
);

===Insert with conflicting IDs===
insert into machine_group(id, area, number)
values (6, 6, 6),
(7, 6, 7),
(8, 6, 8),
(9, 6, 9),
(10, 6, 10)
on conflict (id) do nothing;

==UPDATE==
===REGEXP and REGEXP_REPLACE===
-- replace
UPDATE mytable
SET remark = REPLACE(remark, '&', '&')
WHERE remark LIKE '%&%'

-- remove _|_ -> linebreak \n=10
UPDATE mytable
SET remark = REPLACE(remark, '_|_', chr(10) )
WHERE remark LIKE '%_|_%'

===UPDATE via SELECT===
update single column from another table
UPDATE q_user
SET card_no = q_upload_user.card_no
FROM q_upload_user
WHERE q_user.id = q_upload_user.id

using WITH temp table
WITH map_name_id AS
(
SELECT short_name, min(id) min_id
FROM q_upload_organization
GROUP BY short_name
)
UPDATE q_upload_user2
SET department_id = map_name_id.min_id
FROM map_name_id
WHERE q_upload_user2.dpt_short_name_sap = map_name_id.short_name

sync table tab1 from tab2
UPDATE tab1
SET (group_id, error_de, error_en, error_hu) = (
SELECT group_id, error_de, error_en, error_hu FROM tab2 WHERE tab1.code = tab2.code
)

===pseudonymization===
UPDATE user
SET
username = replace( format('u%7s', id), ' ', '0') -- u0000001 for id=1
,first_name = (array['Max', 'Egon', 'Lisa']) [floor(random() * 3 + 1)]
,last_name = (array['Meier', 'Müller', 'Schmidt')[floor(random() * 3 + 1)]
,email = 'no-mail@schaeffler.com'
WHERE id = 1;

==DELETE==
===Performance for DELETE Statement===
from [https://dba.stackexchange.com/a/35007]
Good: DELETE FROM foo WHERE id IN (select id from rows_to_delete);
Bad: DELETE FROM foo WHERE id NOT IN (select id from rows_to_keep);
Good Alternative for the bad rows_to_keep path:
DELETE FROM foo
WHERE id IN (
SELECT f.id FROM foo f
LEFT JOIN rows_to_keep k ON f.id = k.id
WHERE k.id IS NULL
);

delete based on multiple criteria also in another table
DELETE FROM
table1 t1
WHERE
EXISTS (
SELECT
1
FROM
table2 t2
WHERE
1 = 1
AND t2.date < CURRENT_DATE - 30
AND t1.col1 = t2.col1
AND t1.col2 = t2.col2 )

==CREATE TABLE==
===Create Table with auto-increment id===
CREATE TABLE test_auto_id (
id serial PRIMARY KEY,
value TEXT NOT NULL
);

INSERT INTO test_auto_id (value) VALUES ('asdf');

SELECT * FROM test_auto_id ;

===Create Table Like===
-- without contents
CREATE TABLE myNewTable AS SELECT * FROM myTable WHERE 1=2;
-- without contents but including index etc (but no permissions)
CREATE TABLE myNewTable (like myTable including all);

===Create unlogged cache/caching table===
inspired from [https://martinheinz.dev/blog/105 You Don't Need a Dedicated Cache Service - PostgreSQL as a Cache]
CREATE UNLOGGED TABLE my_cache (
url text UNIQUE NOT NULL,
value jsonb,
delete_at timestamp WITHOUT TIME ZONE DEFAULT now() + '24 hours'::interval );

CREATE INDEX idx_cache_key ON my_cache (url);

INSERT INTO my_cache (url, value)
VALUES ('myURL2.com', '{"key":"value"}');

==FUNCTION in plpgsql==
CREATE OR REPLACE FUNCTION drm.tmp_change_plant_id(id_old int, id_new int)
RETURNS void AS $$
BEGIN

UPDATE myTable SET col = 123 WHERE col = 456;

END;
$$ LANGUAGE plpgsql;
;

==Users==
===grant permission to user===
-- as user myuser
GRANT USAGE ON SCHEMA myschema TO myuser2;
GRANT SELECT, INSERT ON TABLE mytable TO myuser2;

===read only user===
CREATE USER myUser PASSWORD 'myPassword';
GRANT USAGE ON SCHEMA mySchema TO myUser
GRANT SELECT ON ALL TABLES IN SCHEMA mySchema TO myUser

-- needed for views that are added later
ALTER DEFAULT PRIVILEGES IN SCHEMA mySchema GRANT SELECT ON TABLES TO myUser;

-- default to search in another schema
ALTER ROLE myUser SET search_path TO mySchema, public;

-- set default for transactions to read-only
ALTER USER myUser SET default_transaction_read_only = on;

-- check via
SELECT * FROM pg_db_role_setting
-- -> unfold the results

===Disable user===
ALTER USER myuser WITH NOLOGIN

===Change password===
ALTER USER user_name WITH PASSWORD 'new_password';

==Schema Ownership==
GRANT myuser TO postgresadmin;
ALTER SCHEMA myschema OWNER TO myuser;
GRANT ALL ON SCHEMA myschema TO myuser;
REVOKE myuser FROM postgresadmin;

==Other==

===sprintf-alternative===
goal
sprintf 'u%07d', 1
achieved via
SELECT replace( format('u%7s', 1), ' ', '0');

===random item of list===
SELECT (array['Yes', 'No', 'Maybe'])[floor(random() * 3 + 1)];

===Count the occurrences of a substring in a string===
solution based on [https://dba.stackexchange.com/questions/166762/how-do-you-count-the-occurrences-of-an-anchored-string-using-postgresql/166763#166763] and [https://stackoverflow.com/a/42708237]

SELECT
(length(str) - length(replace(str, replacestr, '')) )::int
/ length(replacestr)
FROM ( VALUES
('foobarbaz', 'ba')
) AS t(str, replacestr);
becomes
SELECT
sub1.id,
(length(sub1.str) - length(replace(sub1.str, sub1.replacestr, '')) )::int / length(sub1.replacestr) AS times
FROM
(
SELECT id, checklist AS str, '{"id":'::text AS replacestr FROM d_department
) sub1
which counts the number of (json) ids '{"id":' in a textfield

===Get table column type / describe alternative===
SELECT
table_name,
column_name,
data_type,
character_maximum_length
FROM
INFORMATION_SCHEMA.COLUMNS
WHERE
1 = 1
AND table_schema IN ('schema1','schema2')
AND (table_name LIKE 'c\_%' OR table_name LIKE 'd\_%')
AND column_name = 'order'
ORDER BY
table_name,
column_name

===Concatenate / string_agg columns of multiple rows===
SELECT id, STRING_AGG(c.comment, '
-----
' ORDER BY c.created ASC) AS comment
FROM comment c
GROUP BY id

==Performance Tuning==
use this to check the SQL speed:
EXPLAIN (analyze,buffers,timing)
SELECT * FROM table WHERE ...

===EXISTS===
SELECT 1 WHERE EXISTS ( SELECT 1 FROM table WHERE field = :field )
--Is much faster than performing a COUNT(*) and later check if greater than 0

===UNION vs UNION ALL===
UNION does include a DISTINCT, while UNION ALL does not, so if you know the data is unique, use UNION ALL

===SELECT WHERE IN () Statement===
when passing a long list of values into a statement like
SELECT * FROM myTable where id in (1,2,3,4,5,6,7,8,...);
the statement becomes very slow, see [https://stackoverflow.com/questions/39246746/postgres-query-with-in-is-very-slow]

Better use a WITH clause (=CTE = common table expressions):
WITH value_list AS (
VALUES (1),(2),...
)
SELECT * FROM myTable WHERE id IN (SELECT * FROM value_list);

===Find Missing Index/Indices===
Read https://use-the-index-luke.com/

from https://www.cybertec-postgresql.com/en/index-your-foreign-key/
SELECT c.conrelid::regclass AS "table",
/* list of key column names in order */
string_agg(a.attname, ',' ORDER BY x.n) AS columns,
pg_catalog.pg_size_pretty(
pg_catalog.pg_relation_size(c.conrelid)
) AS size,
c.conname AS constraint,
c.confrelid::regclass AS referenced_table
FROM pg_catalog.pg_constraint c
/* enumerated key column numbers per foreign key */
CROSS JOIN LATERAL
unnest(c.conkey) WITH ORDINALITY AS x(attnum, n)
/* name for each key column */
JOIN pg_catalog.pg_attribute a
ON a.attnum = x.attnum
AND a.attrelid = c.conrelid
WHERE NOT EXISTS
/* is there a matching index for the constraint? */
(SELECT 1 FROM pg_catalog.pg_index i
WHERE i.indrelid = c.conrelid
/* it must not be a partial index */
AND i.indpred IS NULL
/* the first index columns must be the same as the
key columns, but order doesn't matter */
AND (i.indkey::smallint[])[0:cardinality(c.conkey)-1]
OPERATOR(pg_catalog.@>) c.conkey)
AND c.contype = 'f'
GROUP BY c.conrelid, c.conname, c.confrelid
ORDER BY pg_catalog.pg_relation_size(c.conrelid) DESC;

==Admin==
SELECT version();

===PostgreSQL Monitoring Cheatsheet===
see [https://russ.garrett.co.uk/2015/10/02/postgres-monitoring-cheatsheet/]

===Disconnect all sessions===
SELECT pg_terminate_backend(pg_stat_activity.pid)
FROM pg_stat_activity
WHERE pg_stat_activity.datname = 'myDB'
AND pid <> pg_backend_pid();

===Drop and recreate DB via psql ===
SET PGPASSWORD=myPassword
psql --port=5432 --username=myUser
DROP database IF EXISTS myDB;
CREATE database myDB;
\c myDB;
DROP SCHEMA IF EXISTS public;
CREATE SCHEMA public;
\q

===Table size including Indices===
SELECT
table_name,
pg_size_pretty(table_size) AS table_size,
pg_size_pretty(indexes_size) AS indexes_size,
pg_size_pretty(total_size) AS total_size
FROM (
SELECT
table_name,
pg_table_size(table_name) AS table_size,
pg_indexes_size(table_name) AS indexes_size,
pg_total_relation_size(table_name) AS total_size
FROM (
SELECT ('"' || table_schema || '"."' || table_name || '"') AS table_name
FROM information_schema.tables
) AS all_tables
ORDER BY total_size DESC
) AS pretty_sizes;

===Find table column candidates for type enum (less than 10 different values)===
DO $$
DECLARE
r RECORD;
query TEXT;
all_queries TEXT := '';
my_table_name TEXT := 'd_template';
BEGIN
FOR r IN
SELECT column_name, data_type
FROM information_schema.COLUMNS
WHERE table_name = my_table_name
AND data_type NOT IN ('bigint', 'integer', 'timestamp without time zone', 'boolean', 'USER-DEFINED')
ORDER BY column_name
LOOP
query := 'SELECT ''' || r.column_name || ''' AS column_name, count(*) FROM (SELECT DISTINCT ' || r.column_name || ' FROM ' || my_table_name || ' LIMIT 10) s1 HAVING count(*)<10';
--RAISE NOTICE '%', query;
IF all_queries <> '' THEN
all_queries := all_queries || ' UNION ALL ';
END IF;
all_queries := all_queries || query;
END LOOP;
RAISE NOTICE '%', all_queries;
--EXECUTE all_queries;
END $$;

===Vacuum / Autovacuum===
====Manual====
VACUUM VERBOSE ANALYZE myTable;
or with exclusive lock:
VACUUM FULL VERBOSE ANALYZE myTable;
====AutoVacuum====
SELECT name, setting FROM pg_settings WHERE name='autovacuum';
-- should be on
SELECT * from pg_settings where category like 'Autovacuum';
--overview by
SELECT schemaname, relname, last_autoanalyze, last_analyze, last_autovacuum, last_vacuum, n_live_tup, n_dead_tup,*
FROM "pg_catalog"."pg_stat_all_tables"
WHERE schemaname = 'mySchema'
ORDER BY last_autoanalyze DESC

==Performance Analysis==

====List of currently running Queries====
SELECT pid, datname, usename, state, query, query_start, now()-query_start AS runtime, client_addr, application_name, *
FROM pg_stat_activity
WHERE 1=1
ORDER BY query_start ASC ;

====pg_stat_statements module====
pg_stat_statements module is very nice, as it displays which queries run how often and how long, see for example [https://dba.stackexchange.com/questions/44084/troubleshooting-high-cpu-usage-from-postgres-and-postmaster-services]
 Installation
* postgresql.conf
in : section # - Kernel Resource Usage -
add: shared_preload_libraries = 'pg_stat_statements'
* server parameter 'pg_stat_statements.track' must be set to 'top' or 'all'
* restart PostgreSQL
CREATE EXTENSION pg_stat_statements
(in each DB)
* check in available extensions via
SELECT * FROM pg_available_extensions WHERE name = 'pg_stat_statements'
finally run (recent postgres version)
SELECT
round((total_exec_time / 1000 / 3600)::numeric,2) as total_hours,
round((total_exec_time / 1000 / calls)::numeric,3) as avg_sec,
calls num_calls,
query
FROM pg_stat_statements
WHERE 1=1
AND query != '<insufficient privilege>' -- only by current user
-- AND lower(query) LIKE '%select *%join%'
ORDER BY 2 DESC LIMIT 100;
finally run (old postgres version)
SELECT
round((total_time / 1000 / 3600)::numeric,2) as total_hours,
round((total_time / 1000 / calls)::numeric,3) as avg_sec,
calls num_calls,
query
FROM pg_stat_statements
WHERE 1=1
AND query != '<insufficient privilege>' -- only by current user
-- AND lower(query) LIKE '%select *%join%'
ORDER BY 2 DESC LIMIT 100;
* Grant access to non-admin users:
GRANT USAGE ON SCHEMA public TO myUser;
GRANT SELECT ON pg_stat_statements TO myUser;
* Reset stats (as admin user)
SELECT pg_stat_statements_reset()

====Check/analyze why a certain statement runs slow====
from [https://dba.stackexchange.com/a/226481]
EXPLAIN (analyze,buffers,timing)
DELETE FROM myTable WHERE id IN (
SELECT id FROM myTableToDelete
) AND id > 2150
AND id < 2200 ;

EXPLAIN ANALYZE
SELECT id FROM myTableToDelete

analyze via
psql -qAt -f explain.sql > analyze.json
http://tatiyants.com/pev/#/plans/new

====Find and Delete Locks / Processes====
GRANT pg_signal_backend TO postgresadmin WITH ADMIN OPTION;
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE pid <> pg_backend_pid();

-- Display PIDs and Queries running on a table, from [https://jaketrent.com/post/find-kill-locks-postgres/]
SELECT pid, state, usename, query, query_start
FROM pg_stat_activity
WHERE pid IN (
SELECT pid FROM pg_locks l
JOIN pg_class t ON l.relation = t.oid AND t.relkind = 'r'
WHERE t.relname = 'myTable'
);

-- show all processes on current DB
SELECT datname, pid, state, query, age(clock_timestamp(), query_start) AS age
FROM pg_stat_activity
WHERE 1=1
AND state <> 'idle'
AND query NOT LIKE '% FROM pg_stat_activity %'
AND datname = current_database()
ORDER BY age;

-- show locks
SELECT t.relname, *
FROM pg_locks l
JOIN pg_class t ON l.relation = t.oid AND t.relkind = 'r'
WHERE t.relname != 'pg_class';

-- cancel process
SELECT pg_cancel_backend(<my_pid>);
COMMIT;

-- force cancel process
SELECT pg_terminate_backend(<my_pid>);
COMMIT;

-- force cancel all processes
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE pid <> pg_backend_pid() AND datname = current_database();
COMMIT;

===Backup and Restore===
Backup
SET PGPASSWORD=myPassword
pg_dump.exe --host=localhost --port=5432 --username=myUser --dbname=myDB --schema=mySchema --Format c --blobs --verbose --file myDB.dmp
Restore (as admin user postgres)
ALTER SCHEMA "myDB" RENAME TO "myDB.old";
SET PGPASSWORD=myPassword
pg_restore --host=localhost --port=5432 --username=postgres --create --dbname myDB --verbose --exit-on-error myDB.dmp
DROP SCHEMA "myDB.old" CASCADE;

====Export and import single table====
SET PGPASSWORD=myPassword

# export single report table
# use of pg_dump / restore is problematic, since it requires to have the same schema at the target DB, hence using COPY instead
psql.exe --host=myHost1 --port=5432 --username=myUser --dbname=myDB --schema=mySchema ^
--command="COPY myTable TO STDOUT DELIMITER '|' CSV HEADER;" > myTable.csv
# or
psql.exe --host=myHost1 --port=5432 --username=myUser --dbname=myDB --schema=mySchema ^
--command="COPY (SELECT id,name FROM myTable) TO STDOUT DELIMITER '|' CSV HEADER;" > myTable.csv

# import using COPY
psql.exe
--host=myHost1 --port=5432 --username=myUser --dbname=myDB ^
-v ON_ERROR_STOP=1
--command="TRUNCATE TABLE myTable; COPY myTable FROM STDIN DELIMITER '|' CSV HEADER;" < myTable.csv

Visual Studio Code

2025-05-18T15:35:36Z

Torben: /* Draw.io */

[[Category:Coding]]
===settings.json===
// VS Code Core
"diffEditor.ignoreTrimWhitespace": false,
// "editor.fontSize": 18,
"editor.accessibilitySupport": "off",
"editor.formatOnSave": true,
"editor.minimap.enabled": false,
"editor.suggestSelection": "first",
"editor.wordWrap": "on",
"files.eol": "\n",
"files.insertFinalNewline": true,
"files.trimTrailingWhitespace": true,
"files.exclude": {
"**/node_modules": true,
}, "outline.showVariables": false,
"security.workspace.trust.untrustedFiles": "open",
"telemetry.telemetryLevel": "off",
"update.showReleaseNotes": false,
"vsintellicode.modify.editor.suggestSelection": "automaticallyOverrodeDefaultValue",
"window.zoomLevel": 1,
"workbench.startupEditor": "none",

===Shortcuts===
====Log File Cleanup====
* mark text to remove from the log
* to set cursor to all matching lines: CTRL+F2 / CMD+F2
* to delete these lines: CTRL+SHIFT+K / CMD+SHIFT+K

===Extensions===
====Arduiono====
* [https://marketplace.visualstudio.com/items?itemName=vsciot-vscode.vscode-arduino Arduino]
* [https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools C/C++]

settings.json
// Arduino / C
"arduino.ignoreBoards": [
"Adafruit HUZZAH ESP8266"
],
// "C_Cpp.updateChannel": "Insiders",
"[c]": {
"editor.wordBasedSuggestions": false,
"editor.suggest.insertMode": "replace",
"editor.semanticHighlighting.enabled": true,
"editor.quickSuggestions": {
"comments": "on",
"strings": "on",
"other": "on"
}
},

====Better Comments====
* [https://marketplace.visualstudio.com/items/?itemName=aaron-bond.better-comments Better Comments

====Clipboard History====
* [https://marketplace.visualstudio.com/items?itemName=Anjali.clipboard-history Clipboard History]

====CSpell - Code Spell Checker====
* [https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker CSpell]
* [https://marketplace.visualstudio.com/items?itemName=streetsidesoftware.code-spell-checker-german CSpell German]

settings.json
// cSpell
"cSpell.language": "en-US,de-DE",
"cSpell.userWords": [
"Menke",
"Torben"
],
and 
cspell.json (in project dir)
{
"useGitignore": true,
"dictionaries": [
"cspell-words"
],
"dictionaryDefinitions": [
{
"name": "cspell-words",
"path": "cspell-words.txt",
"addWords": true
}
],
"language": "en-US, de-DE",
"ignorePaths": [
"node_modules/**",
"src/assets/i18n/**",
"*.svg"
]
}

cspell-words.txt (configured above)
Torben
Menke
!forbiddenword

comments:
ignoreWords -> forbitten words to allow
dictionaryDefinitions -> custom wordlist/dictionary
ignoreRegExpList -> code blocks to ignore

ignore specific lines. Inside source code files use
// cspell:disable-line -- disables checking for the current line.
// cspell:disable-next-line -- disables checking till the end of the next line.

====Check for unknown words in all files====
cspell --words-only --unique "**/*.{txt,md,html,js,ts,json,yml,py}" > cspell-unknown-words.txt

====CSV/TSV====
* [https://marketplace.visualstudio.com/items/?itemName=mechatroner.rainbow-csv Rainbow CSV]

====Draw.io====
* [https://marketplace.visualstudio.com/items?itemName=hediet.vscode-drawio Draw.io Integration]

====Excel Viewer====
* [https://marketplace.visualstudio.com/items?itemName=GrapeCity.gc-excelviewer Excel Viewer]

====Git====
* [https://marketplace.visualstudio.com/items?itemName=donjayamanne.githistory Git History]
* [https://marketplace.visualstudio.com/items?itemName=waderyan.gitblame Git Blame]

settings.json
// Git
"git.autofetch": true,
"git.confirmSync": false,
"git.enableSmartCommit": true,

====Gnuplot====
* [https://marketplace.visualstudio.com/items?itemName=mammothb.gnuplot Gnuplot]

====HTML====
* [https://marketplace.visualstudio.com/items?itemName=HTMLHint.vscode-htmlhint HTMLHint]

settings.json
// HTML
"html.format.wrapLineLength": 0,
"html.format.maxPreserveNewLines": 2,
"html.format.preserveNewLines": false,
"html.format.extraLiners": "head, body, /html, p, h1, h2, h3, h4, h5, h6",

====JavaScript and Vue====
see https://vueschool.io/articles/vuejs-tutorials/eslint-and-prettier-with-vite-and-vue-js-3/
* [https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint ESLint]
* [https://marketplace.visualstudio.com/items?itemName=esbenp.prettier-vscode Prettier]
=====Vue.js=====
* [https://marketplace.visualstudio.com/items?itemName=Vue.volar Vue]
* [https://marketplace.visualstudio.com/items?itemName=Vue.vscode-typescript-vue-plugin Vue TS]
* [https://marketplace.visualstudio.com/items?itemName=vuetifyjs.vuetify-vscode Vuetify]
* [https://marketplace.visualstudio.com/items?itemName=ZixuanChen.vitest-explorer Vitest]

settings.json
//
// JSON & YAML
//
"[json]": {
"editor.defaultFormatter": "esbenp.prettier-vscode"
},
"[jsonc]": {
"editor.defaultFormatter": "esbenp.prettier-vscode"
},
"[yaml]": {
"editor.defaultFormatter": "esbenp.prettier-vscode"
},
//
// JavaScript & Vue
//
"[javascript]": {
"editor.codeActionsOnSave": [
"source.formatDocument",
"source.fixAll.eslint"
],
"editor.defaultFormatter": "esbenp.prettier-vscode",
// runs with "source.formatDocument"
"editor.formatOnSave": true
},
"[vue]": {
"editor.codeActionsOnSave": [
"source.formatDocument",
"source.fixAll.eslint"
],
"editor.defaultFormatter": "esbenp.prettier-vscode",
// runs with "source.formatDocument"
"editor.formatOnSave": true
},
// Prettier settings - runs with "source.formatDocument"
"prettier.enable": true,
"prettier.enableDebugLogs": true,
"prettier.trailingComma": "all",
// ESLint settings - runs with "source.fixAll.eslint"
"eslint.debug": false,
"eslint.format.enable": true,
"eslint.probe": ["javascript", "typescript", "vue", "html"],
"eslint.validate": ["javascript", "typescript", "vue", "html"],

create package.json
npm init
install packages
npm install --save-dev eslint
npm install --save-dev eslint-config-google
npm install --save-dev prettier
npm install --save-dev eslint-config-prettier
create .eslintrc.json
npm init @eslint/config

.eslint.js
module.exports = {
env: {
browser: true,
es2021: true,
},
extends: [
"google",
"plugin:vue/vue3-recommended",
"plugin:vitest/recommended",
"prettier", // as last one
],
plugins: ["vitest"],
rules: {
// quotes: ["error", "double"],
"vue/valid-attribute-name": "error",
"vitest/max-nested-describe": [
"error",
{
max: 3,
},
],
},
};

.prettierrc
{
"arrowParens": "avoid",
"endOfLine": "lf",
"htmlWhitespaceSensitivity": "css",
"printWidth": 80,
"semi": true,
"singleQuote": false,
"tabWidth": 2,
"trailingComma": "all",
"vueIndentScriptAndStyle": false
}

====LaTeX====
* [https://marketplace.visualstudio.com/items?itemName=James-Yu.latex-workshop LaTeX Workshop]

settings.json
// LaTeX
"latex-workshop.latex.autoBuild.run": "never",
"settingsSync.ignoredSettings": [],
"[latex]": {
"editor.autoIndent": "none",
"editor.formatOnSave": false
// "auto-format": false
},

====Markdown====
* [https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one Markdown All in One]
* [https://marketplace.visualstudio.com/items?itemName=shd101wyy.markdown-preview-enhanced Markdown Preview Enhanced]
* [https://marketplace.visualstudio.com/items?itemName=DavidAnson.vscode-markdownlint Markdown Lint]
settings.json
"[markdown]": {
"editor.formatOnSave": true,
"editor.formatOnPaste": true,
"editor.defaultFormatter": "DavidAnson.vscode-markdownlint"
},

====Markdown Preview Mermaid Support====
* [https://marketplace.visualstudio.com/items/?itemName=bierner.markdown-mermaid Markdown Preview Mermaid Support]

====Perl====
* [https://marketplace.visualstudio.com/items?itemName=sfodje.perltidy perltidy]
Old:
* Perl Toolbox (for coding style check) (requires perlcritic])
* Perl (requires CPAN [https://metacpan.org/pod/distribution/Perl-Tidy/lib/Perl/Tidy.pod Perl::Tidy] and [https://sourceforge.net/projects/ctags/ ctags]
* Perl Debug (requires CPAN [https://metacpan.org/pod/PadWalker PadWalker])

Requirements
* cpan install Perl::Tidy
* cpan install Perl::Critic
* [https://sourceforge.net/projects/ctags/ ctags]

settings.json
// Perl
"perltidy.additionalArguments": [ // see http://perltidy.sourceforge.net/perltidy.html
"--perl-best-practices", // -pbp abbreviation for the parameters in the book Perl Best Practices by Damian Conway
"--blank-lines-before-subs=2", // -blbs
"--closing-side-comments", // adds "## end sub subname etc"
"--indent-columns=2", // -ic does not work in vis.stutio code , but editor.tabSize does
"--noblanks-before-comments", // -nbbc , opposite: --blanks-before-comments=2 / -bbc
// "--maximum-line-length=0", // -l , 0 = infinite
// "--standard-output", // -so
// "--quiet", // -q Deactivate error messages and syntax checking (for running under an editor)
// //
// "--add-whitespace", // -aws add white spaces, use -naws if you do not want any whitespace added, but are willing to have some whitespace deleted
// "--brace-tightness=0", // -bt braces thickness
// "--cuddled-else", // -ce
// "--delete-old-whitespace", // -dws default
// "--ignore-side-comment-lengths", // comments are allowed to be longer than the lines of code
// "--keep-old-blank-lines=1", //-kbl, 0=ignore, 1=stable, 2=keep
// "--maximum-consecutive-blank-lines=1", // -mbl
// "--nohanging-side-comments",
// "--opening-brace-always-on-right", // -bar
// "--paren-tightness=0", // -pt parentheses thickness
// "--square-bracket-tightness=0", // -sbt square brackets thickness
// // "--nohanging-side-comments",
],
"perl-toolbox.lint.excludedPolicies": [
"ErrorHandling::RequireCarping", // die used instead of croak
"InputOutput::RequireCheckedSyscalls", // Lint: BRUTAL: Return value of flagged function ignored - print
"NamingConventions::Capitalization", // var. names all capital or all lower case
"ProhibitEmptyQuotes", // don't complain if using $s =
"ProhibitEnumeratedClasses", // don't complain about [a-z]
"ProhibitEscapedMetacharacters", // don't complain about \d in reg exp / Use character classes for literal metachars instead of escapes
"ProhibitMagicNumbers", // don't complain $n = 14
"ProhibitManyArgs", // don't care how many arguments a function has
"ProhibitNoisyQuotes", // don't complain if using 1 char strings: $s = 'a'
"ProhibitParensWithBuiltins", // don't complain if using () for buildin functions like open
"ProhibitPostfixControls", // don't complain for $n ++ if <something>
"ProhibitPunctuationVars", // don't complain if using $!
"ProhibitUnusualDelimiters", // don't complain about m!^(https?)://!
"ProhibitUselessTopic", // don't complain about use of $_
"RequireDotMatchAnything", // don't complain Regular expression without "/s" flag
"RequireExplicitPackage", // don't require package as first line
"RequireExtendedFormatting", // don't complain Regular expression without "/x" flag
"RequireLineBoundaryMatching", // don't complain Regular expression without "/s" flag
"RequireNumberSeparators", // don't require 141_234_397.0145 instead of 141234397.0145 .
"RequireTidyCode", // don't complain on first line if not all of Perl::Tidy is met
"RequireVersionVar", // don't complain on No package-scoped "$VERSION" variable found
"ValuesAndExpressions::ProhibitInterpolationOfLiterals", // check use of ' vs "
],
// "perl.ctagsPath": "c:\\Users\\t\\Progs\\ctags\\ctags.exe",

====PHP====
* [https://marketplace.visualstudio.com/items?itemName=bmewburn.vscode-intelephense-client PHP Intelephense]

settings.json
// PHP
"intelephense.environment.phpVersion": "7.4.30",

====Python====
see [[Python#Editor:_Visual_Studio_Code]]

==== Remote - SSH ====
* Connecte to remote machine via [https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh Remote - SSH]
* F1
* "Remote SSH: Connect to Host"
* user@server.com

====SQL====
=====Format SQL=====
* [https://marketplace.visualstudio.com/items?itemName=inferrinizzard.prettier-sql-vscode Prettier-SQL]
"Prettier-SQL.SQLFlavourOverride": "postgresql",
"Prettier-SQL.ignoreTabSettings": true

=====SQLite=====
* [https://marketplace.visualstudio.com/items/?itemName=alexcvzz.vscode-sqlite SQLite]

====Shell Format====
* [https://marketplace.visualstudio.com/items?itemName=foxundermoon.shell-format Shell Format]

====Text Linting====
* [https://marketplace.visualstudio.com/items?itemName=PatrykPeszko.vscode-proselint Proselint]

IOS iPhone iPad Apps

2025-05-02T05:12:08Z

Torben: /* Reisen */

[[Category:Software]][[Category:Apple]]
===Zocken===
* [https://apps.apple.com/app/id1465731199 2 Player Games: the Challenge] (2€ to remove ads)
* [https://apps.apple.com/app/id1635978552 1 2 3 4 Player Games]
* [https://apps.apple.com/app/id1596736236 Angry Birds Classic]
* [https://apps.apple.com/app/id1145275343 Super Mario Run]
* [https://apps.apple.com/app/id1235863443 Civi 6]
* [https://apps.apple.com/app/id479516143 Minecraft]
* [https://apps.apple.com/app/id1492005122 Diablo Immortal]
* [https://apps.apple.com/app/id1289067019 Krabat]
* [https://apps.apple.com/app/id1291427111 Evoland 2]
* [https://apps.apple.com/app/id1309032833 Zombie Defence]
* [https://apps.apple.com/app/id1476593876 Table Top Racing]
* [https://apps.apple.com/app/id1406710800 Stardew Valley]
* [https://apps.apple.com/app/id1496786244 BER Bausimulator (Postillion)]
* [https://apps.apple.com/app/id808296431 Crashlands]
* [https://apps.apple.com/app/id979917622 Desktop Dungeons]
* [https://apps.apple.com/app/id552039496 The Room 1]
* [https://apps.apple.com/app/id1496354836 Woodoku]

===Apps===
* [https://apps.apple.com/app/id986420993 WarnWetter]
* [https://apps.apple.com/app/id714464092 Windows Remote Desktop Client]
* [https://apps.apple.com/app/id1435127111 KeePassium]
===Camera===
* [https://apps.apple.com/app/id975925059 MS Lens]
* [https://apps.apple.com/app/id489473238 power Barcode Scanner]
===Orga===
* [https://apps.apple.com/app/id1029207872 Google Keep / Notizen]
* [https://apps.apple.com/app/id293561396 Remember the milk]
* [https://apps.apple.com/app/id1477376905 GitHub]
* [https://apps.apple.com/app/id963034692 Streaks]
===Natur===
* [https://apps.apple.com/app/id1297860122 Flora Incognita]
* [https://apps.apple.com/app/id773457673 Merlin Bird ID by Cornell Lab]

===Reisen===
* [https://apps.apple.com/app/id585027354 Google Maps]
* [https://apps.apple.com/app/id343555245 DB Navigator]
* [https://apps.apple.com/app/id1458716890 Stellarium]
* [https://apps.apple.com/app/id430946556 Park4Night]
* OpenStreetmap (OSM) Contributing
** [https://apps.apple.com/app/id1621945342 Every-Door]
** [https://apps.apple.com/app/id592990211 Go Map!!]

===Sport===
* [https://apps.apple.com/app/id503519713 Zombies Run]
* [https://apps.apple.com/app/id447374873 Komoot]
* [https://apps.apple.com/app/id1255964203 Tabata Timer]
* [https://apps.apple.com/app/id1328791384 Useful knots]
===Musik===
* [https://apps.apple.com/app/id343555245 Deezer]
* [https://apps.apple.com/app/id1482898319 Deezer Audiobooks]
* [https://apps.apple.com/app/id324684580 Spotify]
===Kinder===
* [https://apps.apple.com/app/id1180554775 Anton]
* [https://apps.apple.com/app/id1209287132 Mathematische Tricks]
* [https://apps.apple.com/app/id1486159728 Lego Anleitungen]
===Lesen===
* [https://apps.apple.com/app/id324715238 Wikipedia]
* [https://apps.apple.com/app/id997079563 Kiwix Wilipedia Offline]
* [https://apps.apple.com/app/id944944881 Tolino]
* [https://apps.apple.com/app/id422554835 Onleihe]
===Schauen===
* [https://apps.apple.com/app/id544007664 YouTube]
* [https://apps.apple.com/app/id650377962 VLC]
* [https://apps.apple.com/app/id981496660 ARD Mediathek]
* [https://apps.apple.com/app/id437025413 ZDF Mediathek]
* [https://apps.apple.com/app/id932310976 die Maus]
* [https://apps.apple.com/app/id1023688500 der Elefant]
* [https://apps.apple.com/app/id342792525 IMDB]
* [https://apps.apple.com/app/id545519333 Amazon Prime Video]
===Office===
* [https://apps.apple.com/app/id541164041 MS Office]
* [https://apps.apple.com/app/id842842640 Google Docs]
* [https://apps.apple.com/app/id842849113 Goodle Sheets/Tabellen]
* [https://apps.apple.com/app/id414706506 Google Translator]
===Shopping===
* [https://apps.apple.com/app/id382596778 Kleinanzeigen]
===Chat===
* [https://apps.apple.com/app/id874139669 Signal]
* [https://apps.apple.com/app/id686449807 Telegram]
* [https://apps.apple.com/app/id578665578 Threema]

Wiki Maintainance

2025-05-01T04:57:13Z

Torben: /* How to upgrade this wiki to latest version */

[[Category:Webserver]]
===How to upgrade this wiki to latest version===
* cleanup DB (see below)
* backup DB
* download latest LTS [https://www.mediawiki.org/wiki/Download version]
* extract next to old version (new dir, do not overwrite)
* copy LocalSettings.php
* copy pix/ dir for logo
* download the custom extensions used in LocalSettings.php
** [https://www.mediawiki.org/wiki/Extension:Matomo Matomo]
** [https://www.mediawiki.org/wiki/Special:ExtensionDistributor/HitCounters HitCounters]
** [https://www.mediawiki.org/wiki/Special:ExtensionDistributor/NewestPages NewestPages]
* as long as there are no file attachments in here, that part can be skipped
* adjust wickie symlink
* restart of php might be required via: uberspace tools restart php

===Delete old page revisions/versions/history===
See [https://www.mediawiki.org/wiki/Manual:DeleteOldRevisions.php]
cd maintenance
php deleteOldRevisions.php --delete
php deleteArchivedRevisions.php --delete

JavaScript

2025-03-24T06:00:56Z

Torben: /* ignore Test Coverage for missing else etc */

[[Category:Coding]]
===Basics===
Load after html loading via defer
<script src="main.js" defer></script>

====HTML Form Elements====
Function of button click
<button id="submit" onclick="myFunction()">Anmelden</button>

Function on enter
<input type="number" id="threshold" name="threshold" min="50" max="500" step="1" value="300">
var input_threshold = document.getElementById("threshold");
input_threshold.addEventListener("keydown", function (e) {
// Enter is pressed
if (e.keyCode === 13) {
event.preventDefault();
myFunction();
}
}, false);

===Variables===
Declaration
const var = 123; // constant: no change possible
let var = 123; // block-bound, perferred to var
var var = 123; // old style, better use let instead

Clone/Copy Object
// from https://www.samanthaming.com/tidbits/70-3-ways-to-clone-objects/
const myObject2 = Object.assign({}, myObject);

====Arrays====
Loop over all elements
var rows = table.getRows();
for (var i = 0; i < rows.length; i++) {
var row = rows[i];
...
}

Loop over array of objects
for (let i = 0; i < Object.keys(array).length; i++) {
const key = Object.keys(array)[i];
const value = array[key];
}
NOTE: do not use
for (const key in array) { ... }
as this results in this problem
"The body of a for-in should be wrapped in an if statement to filter unwanted properties from the prototype"

Element in Array
if(myArray.indexOf(myItem) > -1) {...}

Last element of array
my_data.slice(-1)[0]
or
my_data.slice(-1)[0]["myKey"]

=====Array of Objects=====
delete some object properties
// delete not needed object properties
const allowedKeys = new Set(["type", "name", "x_date", "x_url", ...Object.keys(measures)]);
data.forEach((obj) => {
Object.entries(obj).forEach(([key]) => {
if (!allowedKeys.has(key)) {
delete obj[key];
}
});
// extract year from x_date and add as property
obj.year = parseInt(obj.x_date.substr(0, 4));
});

Extract distinct list/set list for property 'type' values
const act_types = [...new Set(data.map((obj) => obj.type))].sort();

Sort by object property "measure" DESC
data = data.sort((a, b) => b[measure] - a[measure]);

===Async fetching via JQuery===
in HTML
<script src="lib/jquery-3.5.0.min.js"></script>
in JS
// array of promises for async fetching
const promises = [];

// ref dictionary to be fetched: Country Code -> Country Name
var mapCountryNames = {};

// fetch countries-latest-all.json containing country reference data like code and continent
function fetch_mapRefCountryData(mapCountryNames) {
const url =
"https://entorb.net/COVID-19-coronavirus/data/int/countries-latest-all.json";
return $.getJSON(url, function (data) {
console.log("success: mapCountryNames");
})
.done(function (data) {
console.log("done: mapCountryNames");
$.each(data, function (key, val) {
mapCountryNames[data[key].Code] = data[key].Country;
});
})
.fail(function () {
console.log("fail: mapCountryNames");
});
}

// Start the async fetching
promises.push(fetch_mapRefCountryData(mapCountryNames, mapContinentCountries));

// Wait for all async promises to be done (all data is fetched), then print message
Promise.all(promises).then(function () {
console.log("All data fetched");
});

==Date handling==
// calculate date via offset
const daysOffset = 7;
const s_data_last_date = "2020-04-01"
const ts_last_date = Date.parse(s_data_last_date)
var minDate = new Date(ts_last_date);
minDate.setDate(minDate.getDate() + daysOffset);

==Helper Functions==
from [https://love2dev.com/blog/javascript-remove-from-array/
function arrayRemove(arr, value) {
return arr.filter(function (ele) { return ele != value; });
}

from [https://stackoverflow.com/posts/3364546/timeline]
function removeAllOptionsFromSelect(select) {
var i, L = select.options.length - 1;
for (i = L; i >= 0; i--) {
select.remove(i);
}
}

// Formats value "Something_Is_HERE" to "Something is here" like sentence
// value: The value to format
// separator: the separator string between words
function formatValueToSentenceLike(value, separator) {
const allLowerCaseValue = value.split(separator).join(" ").toLowerCase();
return allLowerCaseValue[0].toUpperCase() + allLowerCaseValue.substr(1);
}

// modifies array of objects by removing if value == keys
function arrayRemoveValueTextPairByValue(arr, key) {
for (let i = arr.length - 1; i >= 0; i--) {
if (arr[i].value == key) { arr.splice(i, 1); }
}
}

==Local Storage==
// read browser's local storage for last session data
localStorageData = window.localStorage.getItem("my_data");
if (localStorageData) {
var my_data = JSON.parse(localStorageData);
} else {
var my_data = [];
}

==Export and Import / Download and Upload data to Variable==
HTML
<a id="downloadAnchor" style="display:none"></a>
<button id="download_data" onclick="download_data()">Export</button> and Import: <input id="upload_data" type="file" onchange="upload_data(this)">
JS
// download data
function download_data() {
const dataStr = "data:text/json;charset=utf-8," + encodeURIComponent(JSON.stringify([settings, data]));
let html_dl_anchor = document.getElementById("downloadAnchor");
html_dl_anchor.setAttribute("href", dataStr);
html_dl_anchor.setAttribute("download", "eta.json");
html_dl_anchor.click();
}

// upload data
function upload_data(input) {
// from https://javascript.info/file
let file = input.files[0];
let reader = new FileReader();
reader.readAsText(file);
reader.onload = function () {
const uploaded_data = JSON.parse(reader.result);
settings = uploaded_data[0];
data = uploaded_data[1];
};
reader.onerror = function () {
console.log(reader.error);
};
}

==Hide/remove HTML elements==
HTML
< div id="text_intro" >...< /div >
JS
function hide_intro() {
// from https://stackoverflow.com/questions/1070760/javascript-href-vs-onclick-for-callback-function-on-hyperlink
const html_text_intro = document.getElementById('text_intro');
html_text_intro.remove();
}

==Internet Explorer Backward Compatibility==
====Error: Object doesn't support property or method 'includes'====
variant 1:

replace:
if(myarray.includes(key)) {
by:
if(myarray.indexOf(key) > -1) {

variant 2: polyfile
if (!String.prototype.includes) {
String.prototype.includes = function(search, start) {
'use strict';

if (search instanceof RegExp) {
throw TypeError('first argument must not be a RegExp');
}
if (start === undefined) { start = 0; }
return this.indexOf(search, start) !== -1;
};
}

==Tabulator==
// wait for tableBuilt event
table.on("tableBuilt", function () {
myUpdateTableFnc();
});

==eCharts==

==Jest Unittests==
install jest and jsdom for testing browser features like window.localStorage
npm install --save-dev jest jest-environment-jsdom
in package.json set
"scripts": {"test": "jest --coverage","testc": "jest --coverage"},
"jest": { "testEnvironment": "jsdom" }
run jest
npm test
npm run testc

In the file I want to test here `helper.js`
var module = module || {};
module.exports = {
zeroPad,
...
}

The testcasefile `helper.test.js`
describe("Testing zeroPad", () => {
const { zeroPad } = require("./helper");
test("1->01", () => {
expect(zeroPad(1, 2)).toEqual("01");
});
});

===test.each===
To test many cases use
describe("zeroPad()", () => {
const { zeroPad } = require("./helper");
const cases = [
// arg1, arg2, expectedResult
[1, 2, "01"],
[6, 3, "006"],
];
test.each(cases)(
"given '%p', '%p' it shall return %p",
(arg1, arg2, expectedResult) => {
expect(zeroPad(arg1, arg2)).toEqual(expectedResult);
}
);
});

===ignore Test Coverage for missing else etc===
place such a comment before the non-relevant line
/* istanbul ignore else */
/* istanbul ignore next */

==Translations / i18n==
const supportedLanguages = ['de', 'en', 'fr']
const langBrowser = navigator.language.slice(0, 2)
const lang = supportedLanguages.includes(langBrowser) ? langBrowser : 'en'
const messages = await import(`./locales/${lang}.json`).catch(() => import('./locales/en.

Video Encoding

2025-03-23T06:10:45Z

Torben:

[[Category:Software]]
===2024===
see [[Video shrinking]] for Handbrake

===Nice Linux tools===
* Avidemux (simple cutting+converting)
* Kdenlive (joining multiple video/audio tracks)
* WinFF frontend for the encoder FFmpeg
* gkt-recordmydesktop (videoformat=ogv ; problems recording an applications sound output)

====ffmpeg====
=====convert gif to mp3=====
from [https://unix.stackexchange.com/questions/40638/how-to-do-i-convert-an-animated-gif-to-an-mp4-or-mv4-on-the-command-line]
ffmpeg -y -i myanimation.gif -movflags faststart -pix_fmt yuv420p -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" myvideo.mp4

===H.264 codec===
you need this packages
libavcodec-unstripped-52
libavdevice-unstripped-52
libavfilter-unstripped-0
libavformat-unstripped-52
libavutil-unstripped-49
libpostproc-unstripped-51
libswscale-unstripped-0
(or newer versions)

===ogv -> H.264===
1. Win FF is a frontend for FFmpeg
useful parameters:
-f mp4 -vcodec libx264 -acodec libfaac -ac 2

2. ffmpeg by hand (bitrate: -b)
ffmpeg -i input.avi -f mp4 -vcodec libx264 -acodec libfaac -ac 2 -b 1000k output.mp4

3. mencoder input.ogm -ovc xvid -oac mp3lame -xvidencopts pass=1 -o output.avi
(without -oac mp3lame for no sound)

===Kdenlive===

====no h.264 output====
[http://ubuntuforums.org/showthread.php?t=1280510]
to overcome this problem, one can add other rendering profiles:
1) click on one of the predefined rendering profiles in the H.264 group
2) click the 'Create new profile' button
3) Give it a name, and replace acodec=libfaac to acodec=libmp3lame
This will then render with mp3+h264

====Making a screengrab using Kdenlive====
[http://www.kdenlive.org/contribution-manual/howto-make-video-tutorial]

gtk-recordmydesktop can be used to record a desktop video in ogv format.

in order to edit this usind kdenlive create a custom profile: (Settings -> Manage Project Profiles)
set the parameters to the ones of the video
Use this new Profile for your Project.

====Audio to Mono====
ac=1 for mono
ab=64k
ar=44100

Children of Morta

2025-03-16T06:14:23Z

Torben: Created page with "Category:Zocken [https://store.steampowered.com/app/330020/Children_of_Morta/ Children of Morta on Steam] ==Mods== ===4 Player=== https://www.nexusmods.com/childrenofmorta/mods/2"

[[Category:Zocken]]

[https://store.steampowered.com/app/330020/Children_of_Morta/ Children of Morta on Steam]

==Mods==
===4 Player===
https://www.nexusmods.com/childrenofmorta/mods/2

HPMOR

2025-03-04T18:23:49Z

Torben: /* Translations */

= Harry Potter and the Methods of Rationality (HPMOR) =
Link collection to Eliezer Yudkowsky's great fanfiction

== Original Story ==
* [https://www.hpmor.com/ Official website of 'Harry Potter and the Methods of Rationality']
* [https://github.com/rrthomas/hpmor/releases/latest/ Revised PDF and eBook (epub and mobi) versions of HPMOR]
* [https://hpmorpodcast.com/ Audiobook/Podcast by Eneasz Brodski] - [https://podcasts.apple.com/de/podcast/podcast-the-methods-of-rationality-podcast/id431784580?i=1000473946812 at Apple] - [https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5mZWVkYnVybmVyLmNvbS9IYXJyeVBvdHRlckFuZFRoZU1ldGhvZHNPZlJhdGlvbmFsaXR5VGhlUG9kY2FzdA/episode/aHR0cHM6Ly9ocG1vcnBvZGNhc3QuY29tLz9wPTYw?sa=X&ved=0CAUQkfYCahcKEwjA44jfuO34AhUAAAAAHQAAAAAQAQ at Google] - [https://open.spotify.com/episode/3up7VkwoJUTNMwaM6j6qvs?si=Gx3LsHH9SY-v5TOYJHrUxw&dl_branch=1 at Spotify] (complete)
* [https://voraces.podbean.com/ Audiobook by Jack Voraces] - [https://podcasts.apple.com/gb/podcast/harry-potter-and-the-methods-of-rationality-audiobook/id1465181848 at Apple] - [https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkLnBvZGJlYW4uY29tL3ZvcmFjZXMvZmVlZC54bWw at Google] - [https://open.spotify.com/show/4MCs8UYpoBlNp4aRfzB3a5 at Spotify] - [https://www.amazon.com/Harry-Potter-Methods-Rationality/dp/B08JJSF4B3 at Audible] (complete)

== Translations ==
* List of translations, see [http://www.hpmor.com/info/ hpmor.com] and [https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_of_Rationality#Translations Wikipedia]
* [https://github.com/entorb/hpmor-de/releases/tag/WorkInProgress German DE translation (Deutsche Übersetzung) of HPMOR in PDF and eBook (epub and mobi)]
* [https://github.com/kamaradclimber/hpmor/releases/latest French translation (traduction française) of HPMOR as PDF and eBook (epub)]
* [https://hpmoraudio.de/ DE Audiobook/Podcast by Tilmann Glücks] - [https://podcasts.apple.com/us/podcast/harry-potter-und-die-methoden-des-rationalismus-der-podcast/id1630675041 at Apple] - [https://podcasts.google.com/feed/aHR0cHM6Ly9ocG1vcmF1ZGlvLmRlLz9mZWVkPXBvZGNhc3Q at Google] - [https://open.spotify.com/show/4XmiF2dq0DMVxLyVwQJOam at Spotify] (in progress)
* [https://www.youtube.com/watch?v=h32Ht-HUbL0&list=PLfgJSXz3-j3aYhWyR3Q5JzcI3h_eibPls DE Audiobook by Tralex HPMOR at YouTube] (in progress)

== My Contributions ==
* Typesetting [https://github.com/entorb/hpmor-de German PDF and eBook of HPMOR] based on 4 German translations
* Improvements to the [https://github.com/rrthomas/hpmor English PDF] and created eBook

== Follow-up stuff ==
Only proceed after reading/listening to the story, as these contain spoilers

* [https://youtube.com/watch?v=aT_q646lJVo YouTube summary]
* [https://www.fanfiction.net/s/11293489/1/A-Crack-Slash-Epilogue Epiloque: A Crack Slash Epilogue by Alexander Wales]
* [http://www.anarchyishyperbole.com/p/significant-digits.html Sequel: Significant Digits by Alexander Davis]
* [https://askwhocastsai.substack.com/s/significant-digits-audiobook AI read audiobook of Significant Digits] [https://podcasts.apple.com/us/podcast/significant-digits-audiobook/id1777030523 at Apple] [https://open.spotify.com/show/1KRTpm0h4ADeoNlCEYpnZI Spotify] (in progress)
* [https://m.fanfiction.net/s/10636246/1/Following-the-Phoenix Alternative end: Following the Phoenix by hezzel, spinning off at chapter 81]

== Supplementary ==
* [https://en.wikipedia.org/wiki/Harry_Potter_and_the_Methods_of_Rationality Wikipedia]
* [https://de.wikipedia.org/wiki/Harry_Potter_and_the_Methods_of_Rationality Wikipedia DE]
* [https://www.reddit.com/r/HPMOR/ Subreddit r/HPMOR]
* [https://hpmor.fandom.com/wiki/Harry_Potter_and_the_Methods_of_Rationality_Wiki Wiki at Fandom]

Feel free to drop me a [https://entorb.net/contact.php?origin=HPMOR message], have fun, ''Torben''

Backup

2025-01-06T18:38:53Z

Torben: /* rsync */

[[Category:Software]][[Category:Linux]][[Category:Windows]]
KEIN BACKUP - KEIN MITLEID

As harddisks are known to break from time to time and re-setting up an old measurement pc is quite some pain in the ass, doing a backup onto an external USB harddisk (40-100€) might be worth thinking about...

Basically we have to distinguish between two types of backups:
# Frequently changed data (your personal data)
# Full image of the PC (important for measurement PCs), that includes all needed drivers, libraries, etc and allows restoring of the PC in no time
The first one should be done very frequently (daily/weekly?), the second one once a year or whenever some greater changes to the PC are done.

Concerning the USB harddisk I prefere small 2.5" ones that work without a separate power supply.

Here I wrote a little guide that might help you. 
Happy backing up Torben (18.02.2010)

==Backup of a single folder==

===Windows===
====Directory Compare====
I like the free tool [http://tp.lc.ehu.es/jma/win32/dirco.html Directory Compare] and will now explain you how I use it to backup a single folder like My Documents (De: Eigene Dateien)

* First get the tool. I prefer the .zip file version, since I see no need in installing it.
* Unpack the zip-file or install the tool into some folder (remember this folder!)
* Run the tool by starting "DirCmp.exe"
* Select the folder you want to backup as "Source" and the one you want to copy it to as "Target" (e.g. your external harddisk/networkdrive etc, but keep in mind that the drive has to be connected using the same driveletter before stating the backup...)
* Click on "Scan" as a test
* Under "Options->Configure->Unattended mode" check "use direct API calls..." to prevent the tool to produce message windows
* Click on File->Save to save this settings into a file (e.g. myBackup) that is placed in the program folder. e.g.: c:\program files\Directory Compare\myBackup
* Leave the program
* Use the windows explorer to navigate to the program folder
* Right click on "DirCmp.exe" -> Create new Shortcut
* Rename it to something nice
* Right click on it -> properties (De: Eigenschaften)
* Edit the link target (De: Ziel) and append "myBackup.dcp /m/q" without the " (/m stands for mirror=make target exactly the same as source, overwriting all changes to files done on taget)
* Now just create a desktop shortcut to this file and you are done
Keep in mind, that all changes to files in the target folder are lost and overwritten with each backup, since /m is set, see above.
If you want to keep an older version of the backup, just rename the folder on your external harddisk, so a new copy of all your data is created at the next backup.

In order to increase security, I strongly advice using two different USB harddisks alternating, e.g. one for even months/weeks, one for uneven ones. The reason is, that if your PC breaks while your USB disk is connected you might loose it...
(This also keeps a little more history of changes)

====Zip Folder====
Using [https://sourceforge.net/projects/unxutils/ UnixUtils] for Windows zip util. Should be possible using windows build-in Powershell (Compress-Archive input.txt output.zip).
@echo off

REM set outputfolder
set BACKUPFOLDER="sicher"

REM get CurrDirName
for %%I in (.) do set CurrDirName=%%~nxI

REM use date and time commands to fetch a datestr in format 180830_0647
REM for English Windows this has to be modified. Currently written for
REM date returning '30.08.2018' and time returning ' 6:47:36,10'
set DATESTR=%date:~-2,2%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
REM replace ' ' in hours <10 with 0
set DATESTR=%DATESTR: =0%

REM mkdir
if not exist "%BACKUPFOLDER%" mkdir "%BACKUPFOLDER%"

REM zipping using UnixUtils' zip, excluding backupfolder
%UserProfile%\Documents\Progs\UnixUtils\zip.exe -9 -r "%BACKUPFOLDER%"\%CurrDirName%_%DATESTR%.zip *.* -x *%BACKUPFOLDER%\*
REM -x *sicher\* => exclude folder "sicher"

====rsync for Windows====
There is a port of rsync for Windows using cygwin, [https://www.heise.de/download/product/rsync-1271 download here].

=====Update of 2021: better use robocopy=====
robocopy SOURCE TARGET /MIR /FFT /Z /R:10 /W:5 /MT:8 /log:robocopy.log /NP /NS /NC
/MIR = Mirror
/MT = MultiThreading (default: 8 threads)
/FFT = Assumes FAT file times (two-second precision).
/Z = Copies files in restartable mode
/R = Retries
/W = Wait time between retries
/NP = log: no progress (%)
/NS = log: no file sizes
/NC = log: no file classes
/NFL = log: no file names
/XD = exclude dirs, does not allow wildcards
/XF = exclude files, allows wildcards: *.tmp

For local Backup on USB disk you should remove the retry mechanism and use only 1 thread
robocopy SOURCE TARGET /MIR /Z /R:0 /W:0 /MT:1 /log:robocopy.log /NP /NS /NC

To move instead of backup
Robocopy ".\source" ".\target" /move /e /MT:8 /R:3 /W:5 /LOG:".\robocopy_move.log" /is /it
/e include all subdirectories
/is Includes the same files. Same files are identical in name, size, times, and all attributes.
/it Includes “tweaked” files. Tweaked files have the same name, size, and times, but different attributes.

=====Example 1 - mini=====
@echo off
set rsync="E:\progs\rsync\rsync.exe"
set source=/cygdrive/"D/photos/2017/"
set target=/cygdrive/"E/backup/photos 2017"
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%
rem a -> rlptgoD
rem --modify-window=2 -> 3602 might help sometimes
rem modify-window=2 -> allow for time differences of 2 sec, as for MS FAT

=====Example 2=====
@echo off
set rsync="C:\Users\menketrb\Documents\Progs\rsync\rsync.exe"
set source=/cygdrive/"C/Users/menketrb/Documents/"
set target=/cygdrive/"U/sicher/doks-KOPIE"
set targetReal="U:\sicher\doks-KOPIE"
set exclude=--exclude='My Music' --exclude='My Pictures' --exclude='My Videos'

REM write current date to file inside source folder
set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
REM replace ' ' in small hours with 0
set DATESTR=%DATESTR: =0%
set datefile=%targetReal%\0backup-date-U-%DATESTR%.txt
date /T >> %datefile%
time /T >> %datefile%

REM set target readonly off and on
attrib -r %targetReal%\*.* /s
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%
attrib +r %targetReal%\*.* /s

=====Example 3 - keep daily, weekly and monthly backups=====
@echo off

set rsync="C:\Users\menketrb\Documents\Progs\rsync\rsync.exe"
set source=/cygdrive/"C/Users/menketrb/Documents/"
set sourceReal="C:\Users\menketrb\Documents"
set target=/cygdrive/"C/Users/menketrb/sicher/doksKOPIE"
set targetReal="C:\Users\menketrb\sicher\doksKOPIE"
set exclude=--exclude=Progs --exclude='My Music' --exclude='My Pictures' --exclude='My Videos'

REM write current date to file inside source folder
set DATESTR=%date:~-2,4%%date:~-7,2%%date:~-10,2%_%time:~0,2%%time:~3,2%
:: replace ' ' in small hours with 0
set DATESTR=%DATESTR: =0%
set datefile=%sourceReal%\0backup-date-Lokal-%DATESTR%.txt
date /T >> %datefile%
time /T >> %datefile%

:: get date in several formats for name of backups
set monat=%date:~-7,2%

:: Get DayOfWeek as number
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get DayOfWeek') DO (
if "!count!" GTR "0" GOTO next
set dow=%%D
SET /a count+=1
)
:: /a = arithmetisch
:next

:: Get WeekInMonth as number
SETLOCAL enabledelayedexpansion
SET /a count=0
FOR /F "skip=1" %%D IN ('wmic path win32_localtime get WeekInMonth') DO (
if "!count!" GTR "0" GOTO next
set wim=%%D
SET /a count+=1
)
:: /a = arithmetisch
:next

REM remove write protection
attrib -r %targetReal%\*.* /s

echo =
echo ======= Tagesbackup: rsync-d%dow% ========
echo =
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%/rsync-d%dow%

echo =
echo ======= Wochenbackup: rsync-w%wim% ========
echo =
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%/rsync-w%wim%

echo =
echo ======= Monatsbackup: rsync-m%monat% ========
echo =
%rsync% -ahv --modify-window=2 --no-perms --no-owner --no-group --delete --delete-excluded %source% %target%/rsync-m%monat%

REM set write protection
attrib +r %targetReal%\*.* /s

del %datefile%

===Linux===
====rsync====
Use the command line tool rsync e.g.:
rsync dir1/*.txt dir2/
# delete from target what is not in source
rsync -r --delete dir1/ dir2/

rsync -rvhu --delete --delete-excluded dir1 dir2
rsync is able to speak ssh: (z = compress during transfer)
rsync -rvhuz --no-links --delete --delete-excluded dir1 user@server:dir2

An alternative is [http://rsnapshot.org rsnapshot] [http://wiki.ubuntuusers.de/rsnapshot Gute Deutsche Anleitung] (Thanks to Philipp + Malte)

An alternative with GUI, similar to Apple's TimeMachine is [http://wiki.ubuntuusers.de/Back_In_Time Back In Time]

====rsnapshop====
sudo apt-get install rsnapshot
sudo mkdir /sicher-rsnapshot
in /etc/rsnapshot.conf (use tab to separate key-values pairs!)
# target folder
snapshot_root /sicher-rsnapshot

# retain levels = how many backups to keep per level (alpha, beta, gamma, delta)
# every day
retain alpha 7
# every week
retain beta 4
# every month
retain gamma 12
# every year
retain delta 5

# dirs to backup
# LOCALHOST
backup /home/ localhost/
backup /etc/ localhost/
test configs
rsnapshot configtest
run manually
sudo rsnapshot alpha
run via crontab
sudo crontab -e
# m h dom mon dow command
10 0 * * * /usr/bin/rsnapshot alpha
# every week on Monday
20 0 * * 1 /usr/bin/rsnapshot beta
# every month 1.X.
30 0 1 * * /usr/bin/rsnapshot gamma
# every year 1.1.
40 0 1 1 * /usr/bin/rsnapshot delta

==Full image of a harddisk drive==
I like the free tool [http://www.runtime.org/driveimage-xml.htm DriveImage XML] that be placed on a boot CD in order to do a full backup of a harddisk or partition. It can be found on some bootcds you can find in the net, like UBCD4Win or Hiren's BootCD, but I prefere creating a small personal bootcd without any other tools on it. In order to do so read the next passage or skip it if you already have a suitable boot cd with DriveImage XML on it.

===Build a Boot CD===
For some very strange reasons I was not able to build a working bootcd when running the tools in a virtualbox' WinXP. Using a "real" Windows (Vista 64 in my case) it all worked out fine...
* Fetch [http://www.nu2.nu/pebuilder/#download Bart's PE Builder]
* Ensure you have a WinXP Install CD ready
* Install Bart's PE Builder
* Get the [http://www.runtime.org/driveimage_xml.cab DriveImage XML plugin (.cab file)] for Bart's PE Builder
* Insert your WinXP CD
* Run Bart's PE Builder
* Tell the program where the WinXP install is located (CD Drive)
* Click on plugins and add the previously downloaded DriveImage XML plugin
* Close the plugin window
* Enable the creation of an iso file
* Start building
* Close the program and burn the newly created image onto a cd (not the file, but use the iso-image-mode of your burn program)
Instead of burning directly you might want to try you bootcd first using virtualbox...

===Backing up===
* FIRST: Attach the external hard disk that you want to store the image of your pc on
* Boot from a bootcd that has DriveImage XML installed
* Start the tool DriveImage XML
* Do the backup and ensure that the image is stored on your external harddisk
* Settings: Compression = "Low" makes sense

===Linux and dd, e.g. for SD cards (RasPi)===
from [https://raspberrypi.stackexchange.com/questions/311/how-do-i-backup-my-raspberry-pi]
dd if=/dev/sdX of=/mnt/backup/sdX.dd bs=1M
# (bs = blocksize)

# with on the fly gzip compression
dd if=/dev/sdX | gzip > /mnt/backup/sdX.dd.gz

# with on the fly bzip2 compression (slower, but better compression)
dd if=/dev/sdX | bzip2 > /mnt/backup/sdX.dd.bz2

# with on the fly xz compression (slower, but better compression)
dd if=/dev/sdX | xz > /mnt/backup/sdX.dd.xz

Speed/size comparison for a 16GB SD card with only few GB in use (without filling empty space with zeros, see below)
time bytes MB/s compression
dd 00:43:07 15931539456 5.9 100.0%
gzip 00:43:01 5205027237 1.9 32.7%
bzip2 00:51:47 5114374745 1.6 32.1%
xz 01:01:36 4959001460 1.3 31.1%
after zero-filling of empty space:
gzip 00:43:00 711255877 2,6 4.5%

To improve compression you can fill empty blocks with zeros first on the device, but this takes time and causes write load on the device (maybe bad for ssds)
dd if=/dev/zero of=asdf.txt ; rm asdf.txt
# (assuming your current working dir is on the partition you want to fill the free space with zeros.

To restore the backup, reverse the commands:
dd if=/mnt/backup/sdX.dd of=/dev/sdX bs=1M

# when compressed using gzip
gzip -dc /mnt/backup/sdX.dd.gz | dd of=/dev/sdX bs=1M

# when compressed using bzip2
bunzip2 /mnt/backup/sdX.dd.gz | dd of=/dev/sdX bs=1M

# when compressed using xz
xunz /mnt/backup/sdX.dd.xz | dd of=/dev/sdX bs=1M

==Disaster Recovery==
===Undelete single files===
To enable for recovery, make sure to write as little as possible onto the partition. In Linux unmounting is a good option.
I suggest using the Linux tool photorec, which scans an (unmounted!) partition (ext2/3, Fat32, NTFS, ...) for deleted files and recovers them to another partition.
photorec ships with testdisk, so to install in Ubuntu use
sudo apt-get install testdisk
afterwards run
sudo photorec

===Fix partition table / recover deleted partition===
see [https://www.linuxquestions.org/questions/linux-general-1/how-to-recover-deleted-files-from-a-fat32-volume-in-linux-739511/ TestDisk ]

===securely wipe / overwrite disk===
====Linux/MacOS====
prior to trashing or selling a harddisk you might want to wipe all data
# fill with a zero-filled file
cat /dev/zero > asdf.txt
# or fill whole drive with zeros
dd if=/dev/zero of=/dev/sdX
# or more secure fill with random
dd if=/dev/urandom of=asdf.txt asdf.txt
# or
cat /dev/urandom > asdf.txt
====Windows====
https://www.feyrer.de/g4u/nullfile-1.01_64bit.exe

Mac

2025-01-06T06:10:20Z

Torben: /* Set Hostname */

Video shrinking

2024-12-23T15:02:44Z

Torben:

[[Category:Software]]

===2024===
Use [https://handbrake.fr Handbrake] on Windows, Mac, Linux

Select Preset -> General -> Fast or Very Fast in 1080 or 720

===Linux===
Dank an Johannes!
mencoder -ovc help # zeigt verfügbare Codecs
# lavc ist angeblich der beste. Ab und zu wird auch x264 empfohlen.
mencoder -vf -o output.avi -oac copy -ovc lavc input.avi
# Drehung mit Option rotate=1 / rotate=2

===Windows as of 2018===
Today I prefer using [https://handbrake.fr Handbrake] for video shrinking / compression as it work in Windows, Linux and Mac.

For movies captured by my digital camera or smartphone 
I use preset "Matroska -> H.265 MKV 720p30" as template and modify 
Picture -> Height 720 for downscaling to 720pt 
Video -> Quality 25 for most videos 
Audio -> 64bit Mono (for camera or telephone videos) 
(this modifications can be saved as "User Preset" which can be set to default preset)

Trimming some seconds off the ends is achieved in section "Source" by switching from "Chapters" to "Seconds".

Rotate
On Filters tab choose Rotate

===Windows as of 2010===
I like the free windows tool [http://www.pcfreetime.com/ Format Factory]!

====My Settings====
For compressing the videos taken by my digicam (resolution=640x480px) I usually use the following settings: 
Video
* container: mkv (or mp4)
* codec: H264
* bitrate: between 512 and 1024 kbit/s (512kbit/s for 640x480px seems to be enough) (note: 600MB/hour equals 1365kbit/s)
* 2 Pass encoding: True (for better quality, but much longer computing time)

Audio
* only one channel (=Mono) @ 64kbit/s (as the tiny mic. in the digicam is not able to record real stereo)
* sometimes I even disable the audio channel if there is nothing but noise on it.

Output Folder -> "Output to source file folder"

====Rotating the video by 90°====
use the following settings:
* Rotate = Left or Right
* (AspectRatio=4:3) was needed in earlier versions, now use '''Automatic'''

====Beispiele====
16:9, FullHD: 1920x1080 (oder auch 1280x720) schrumpfen auf 800x450?
Video Size: 800x450
Aspect Ratio: 16:9

====My Format Factory Profiles====
Here are my custom profiles, that can be found at
c:\Users\...\Documents\FormatFactory\VideoCustom\

h264, 512kbit, mono
Type=MP4

VideoCodec=AVC(H264)
VideoBitrate=512000
Width=0
Height=0
FPS=0.000
AspectRatio=0.000

AudioCodec=AAC
AudioBitrate=64000
SampleRate=0
Channel=1
Volume=0

TowPassEncode=1

h264, 512kbit, mono, rotate (one has to set the aspect ratio by hand!)
Type=MP4

VideoCodec=AVC(H264)
VideoBitrate=512000
Width=0
Height=0
FPS=0.000
AspectRatio=1.333

AudioCodec=AAC
AudioBitrate=64000
SampleRate=0
Channel=1
Volume=0

TowPassEncode=1

mute instead of mono:
AudioCodec=AAC
AudioBitrate=0
SampleRate=0
Channel=0
Volume=0
(It turned out, that FF version 2.50 does ignore the Channel=0 setting, so you habe to disable audio in the dialog)

Sentry

2024-12-03T08:01:19Z

Torben: /* Performance Date */

[[Category:Software]][[Category:Coding]][[Category:Python]]
==Python==
import sentry_sdk

sentry_sdk.init(
dsn="https://12345@my-server.com/12",
environment="dev",
# debug=True,
enable_tracing=False,
)
# for self-signed certs
# use browser to export the WHOLE certificate chain to
# ca_certs="myCertBundle.crt",

===Exceptions===
====Var 1: all unhandled Exceptions are sent to Sentry====
# Example
division_by_zero = 1 / 0

====Var 2: custom event====
event_data = {
"message": "My Custom Event Title",
"transaction": "My Transaction",
# "user_id": 123,
# "action": "custom_event",
# "data": {"key1": "value1", "key2": "value2"},
}
with sentry_sdk.push_scope() as scope:
# scope.user = {"id": "user-123"}
event = sentry_sdk.capture_event(event_data)

====Var 3: Forward Handled Exception====
try:
division_by_zero = 1 / 0
except Exception as e:
sentry_sdk.capture_exception(e)

===Performance Date===
# enable tracing
sentry_sdk.init(
...
enable_tracing=True,
)
...
with sentry_sdk.start_transaction(op="task", name="DB"):
# optional spans inside the transaction
# with sentry_sdk.start_span(description="connect"):
# (connection, cursor) = connect()
# with sentry_sdk.start_span(description="execute sql"):
# results = execute_sql(cursor=cursor, sql=sql)
# with sentry_sdk.start_span(description="disconnect"):
# cursor.close()
time.sleep(3)

Add custom tag
# set plant as Sentry tag
scope = sentry_sdk.get_current_scope()
scope.set_tag("plant", my_plant_id)