Problem

While I could use the WebSocket application as a kind of “heartbeat”, there may arise situations where this is not the optimal solution.

For this purpose I run on my machine an application which fetches the latest trades, calculates average price (not average weight price yet) and stores it in a db.

But I need to run that piece of code every minute exactly.

Approach

Under macOS (Big Sur or any other) there’s two ways to define a scheduled application. Pre Big Sur I used the Launch Agents functionality, which requires a plst file with relevant attributes/data. This file is then kind of “loaded” into the launchctl daemon.

That was the first approach and it failed miserably trying to load the plst file.

So I googled a bit and found this solution: https://www.jcchouinard.com/python-automation-with-cron-on-mac/

It pretty simple explains how you can use crontab to do the same. And it actually works.

So, remember “De-complexify problems”, I first did a very simple test. A little Python application which would just print the date. I didn’t even know yet where it would print it (in the system console maybe?)

import time

from datetime import datetime

timestamp = time.time()

print(datetime.fromtimestamp(timestamp)

So I added a first scheduled script to crontab:

* * * * * /Users/michael/Python/script.py>>/Users/Michael/Python/output.log

So I expected to see a an output.log file but I didn’t.

Fortunately, this was new to me, cron sent me a mail on the machine.

There it said: Operation not permitted.

Here the article (see link above) helped. I needed to add cron to the Applications which have Full Disk Access.

Now it worked like a charm.

Next step was to try and have cron run my python script which would fetch every minute data from Bitstamp.

This failed miserably. And it took me hours to figure out what the reason was.

The error message, as before I received emails with the error, said:

ImportError: No module named requests

When I ran the same script manually it worked like a charm. So it looked like there was a problem with the installation of the module requests, at least cron didn’t seem to be able to find that module.

I tried installing in another directory, failed.

I tried uninstalling, reinstalling, it installed in the same directory, failed.

Then I found help via Google search. You can actually prepend the PYTHONPATH to the cron tab command, like this:

* * * * * PYTHONPATH=/usr/local/lib/python3.9/site-package /Users/michael/Python/script.py

Ever since the script runs ever minute and does it’s job: retrieve the trades from the last minute from Bitstamp.

Make sure output is written immediately to log files

As I have several cron jobs running, outputting some data, specially to see they are live and running, I had a problem: those jobs would only print to my out Oles after they had received a certain amount of data. This resulted in a not so chronological output. I had to make sure that the output is sent immediately.

And, alas, there’s a solution: unbuffer!

Install unbuffer with brew

brew install expect

Then issue your calls like:

/usr/local/bin/unbuffer python3 pathToYourScript.py >> pathToYourLog.txt

The paths should be ABSOLUTE for them to work 100%. and it works, output is immediately written.

Another thing I learned and tested already: One can call simply scripts from crontab. This makes crontab way more easy to read and I can put the heavy lines into a simple shell file.

Simplify scheduled jobs

As mentioned above I use crontab on macOS cause on Big Sur somehow the launchd doesn’t load my plist files properly. Others report the same issue.

On crontab it’s crowded. One line can be more than 200 characters long, which makes editing (and even worse viewing) a pain. Reason for that long line is:

I need to add the proper PYTHONPATH, the unbuffer from the last chapter, the proper path to python3, the proper path to my script and finally the proper path to my out file.

And using those shell editors ain’t a lot of fun either.

But with BBEdit on my Mac, my dearest most loved and used companion since I have Macs, editing files is a breeze. So the route is: creating a shell script which does what the cronjob would do, and call from the cronjob the script. Sounds a bit complicated, having a 2 step process now as compared to one step before.

Before

cronjob => python script

Now

cronjob => shell script => python script

The advantage is a much easier interface to maintain my scheduled jobs, cause editing the shell script is way easier than editing the crontab, and crontab is very simplistic now, with only about 90 characters per job.

I had a problem though now: In case I need to “interrupt” my scheduled jobs, how would I achieve this? Turns out I can send any “signal” to my python script. Before I always used KILL only. But this prevented the script from doing clean up work, such as deleting the lock file. By using another signal, INT in this case, I can intercept and react accordingly.

List of available signals:

HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM URG STOP TSTP CONT CHLD TTIN TTOU IO XCPU XFSZ VTALRM PROF WINCH INFO USR1 USR2

INT seems the same as <ctrl><c> which I use if I manually invoke those scripts.

Unfortunately this didn’t seem to work, at first at least.

Python code to act upon:

import signal
signal.signal(signal.SIGINT, receiveSignal)

Use the receiveSignal handler to handle what needs being handled, ie. getting rid of lck files.

Kill won’t let the code react on it, it simply terminates

If you run a python script by cron, will spawn always 2 processes. https://stackoverflow.com/questions/24989154/cronjob-command-starts-2-processes-for-1-python-script

And I didn’t know about that. Well I knew I always had 2 processes, but didn’t know why. Killing one killed the other usually as well.

Now knowing one is the script and the other one is python runtime (can we call it like that?) I just need to send the signal to the proper process.

And I’m doing this with a nice little shell script which ps|grep and finds the process id to kill the appropriate process.

#!/bin/zsh

process=$(ps -ax|grep -v grep|grep “Cellar.*temp_socket.py”)

pid=${process:0:5}

kill -s int $pid

echo “temp_socket.py terminated”

End result:

scripts which send their output immediately to the out file and shell scripts which let me easily terminate those processes without to much of manual work.

Delete lock files upon machine boot/reboot

Giving launchd / launchctl another try

So while I was mostly settled with my setup there was one problem:

I do not know which signal my processes (my bots) receive when I reboot my machine.

Which is why I can’t properly close my bots and let them do the clean up (mainly moving log files and deleting the lock files).

After some thinking I got the idea: I only have to make sure that upon login those lock files are deleted. But crontab / cronjobs wasn’t done for this purpose.

Maybe give launchd / launchctl another try?

-rwxr–r–@ 1 root wheel 599 Feb 2 19:25 deleteLockFiles.plist*

So you need to put a PLIST (not plst, my first attempt was always with plst which did never work) file into the ~/Library/LaunchAgents directory.

Content of PLIST file:

<?xml version=”1.0″ encoding=”UTF-8″?>

<!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN” “http://www.apple.com/DTDs/PropertyList-1.0.dtd”>

<dict>

<key>Label</key>

<string>com.yourName.deleteLckFiles</string>

<key>ProgramArguments</key>

<array>

<string>/path/to/your/script/deleteLckFiles.sh</string>

</array>

<key>KeepAlive</key>

<key>StandardErrorPath</key>

<string>/path/to/your/script/launchd_error.txt</string>

<key>StandardOutPath</key>

<string>/path/to/your/script/launchd_log.txt</string>

<key>RunAtLoad</key>

<true/>

</dict>

</plist>

load the plist file:

sudo launchctl load deleteLckFiles.plist

unload the plist file:

sudo launchctl load deleteLckFiles.plist

And the shell script itself can look like this:

#!/bin/zsh

cd /usually/your/home/directory

ls -d *.lck

rm *.lck

And that’s it.

As always: start slow and easy (remember: Decomplexify problems):

As those background jobs (crontab/launchd/launchctl) do have different “default” paths, this is always a bit a gamble to know where your output files end up and if crontab and launchd can access your scripts

So first:

create a simply script just echoing “Hello world”, have the script and the output file in your home directory, just to make sure.
then you may move the script to the directory where it finally will be called, if this does not work, you may need to give in the Security System Preference Panel Full Disk Access to the shell

and increase the complexity
I gave up trying to run my bots via launchd, couldn’t figure out how to define the paths so the python scripts work properly, but my shell scripts run just about fine via launchd / launchctl

Schedule programs on macOS