Creating a responsive web server
For the visual tool to analyse my charts I needed something to deliver the data to the tool.
While I was a few months back playing around with ideas and technologies, I came across NodeJS. Which is the easiest way for you to create a Webserver based on JavaScript language.
Don’t believe me? Checkout the document Introduction into programming from the ZIP file you find in the description of this video: https://youtu.be/l3undRJAuAU.
There you find information to get started with NodeJS.
Let me tell you: it is so easy, it makes me totally happy to be able to return data for asynchronous calls to my webpage within a few minutes.
var path = require(‘path’)
var express = require(‘express’);
var app = express();
var session = require(‘express-session’)
app.use(session({
secret: ‘chosen language’,
saveUninitialized: true,
resave: false,
}))
var server = app.listen(3000, function (err) {
app.use(express.static(__dirname + ‘/’));
if (err) {
console.log(err.message)
}
console.log(“Server listening at port 3000”)
});
Obviously it does not do much besides listening to incoming requests on port 3000.
You add a little bit of code like:
app.get(‘/’, (request, response) => {
response.sendFile(path.join(__dirname, ‘index.html’));
});
app.get(‘/manual’, (request, response) => {
response.send(“Hello world, no HTML code, pure text”)
});
app.get(‘/test’, (request, response) => {
let path = url.parse(request.url).pathname;
console.log(path)
json = { Name: ‘Michael’, LastName: ‘Zischeck’ }
response.json(json)
})
Then you can call your web server like:
http://localhost:3000/manual
http://localhost:3000/test
http://localhost:3000/ (this will serve you the index.html from the same directory as from where you run the server
Isn’t that simple?
How about returning data from a database, sqlite3 in this example:
app.get(‘/data’, (request, response) => {
let path = url.parse(request.url).pathname;
var parts = url.parse(request.url, true);
var query = parts.query;
action = query.page
console.log(“action:” + action)
if (action == undefined) {
sql = ‘SELECT id, price from tableName limit 0, 1000’;
db.all(sql, (err, rows) => {
if (err) {
console.error(err.message);
}
rows.unshift({ “id”: “id”, “price”: “price” })
response.json(rows);
});
} else if (action == “next”) {
page++
sql = `SELECT id, price from tableName limit ${page * 500}, 1000`;
db.all(sql, (err, rows) => {
if (err) {
console.error(err.message);
}
rows.unshift({ “id”: “id”, “price”: “price” })
response.json(rows);
});
} else if (action == “prev”) {
page–
sql = `SELECT id, price from tableName limit ${page * 500}, 1000`;
db.all(sql, (err, rows) => {
if (err) {
console.error(err.message);
}
rows.unshift({ “id”: “id”, “price”: “price” })
response.json(rows);
});
}
});
That’s it! Complete code for my visual chart analysis tool! How easy is this?
Going back to Python
While I was almost done with my trading bot in NodeJS, I basically converted the most important parts of my Orst attempt in Python, it happened that when I run simulations, like tens of thousands of rows to process, NodeJS went nuts.
I really tried, and I really would have wanted to stay with NodeJS, but I won’t! To much asynchronous.
In Python I have everything under control.
What I will leave in NodeJS is my other project. A tool to analyse XRPL accounts.
The good news though is this:
As I ported from Python to NodeJS, I simplified a lot. Made the whole logic more straightforward, and guess what: I found a simple, very stupid but very powerful bug in my logic!
So even if this sideway to NodeJS and back to Python was just about to figure out this stupid bug, then it was worth it.
Actually it was not even a bug, it was more a strange thing in my logic.
It boils down to this:
With my trading bot I want to prove that even in bearish markets I can profit. So basically I apply the Stop Loss strategy explained here.
In short:
If prices drop, I sell, and I rebuy at lower price, if possible.
Sometimes, the price does not drop further after I sold, but keeps rising. Then I have to rebuy in at a higher price, and at a loss obviously.
That’s all part of the “strategy”, the idea being I can buy more often low, than I have to buy high. And that I can buy relatively lower than I have to buy higher.
But there’s times, hopefully a lot of times, where the prices just keep rising. What then? Well I have to adjust my triggers as to when I want to sell. Those triggers keep rising.
In my old code I would “UP THE ANTE” every time a new high was reached. With the stupid effect, that, obviously, after the high the price starts dropping any my bot does its duly work: it sells (unfortunately at a way to high price).
I figured now that I should better only “UP THE ANTE” a little bit, like a 1/10 of the previous high price and the new high price.
Like old price (for my calculations) was 1$, new price is 1.20$. Now I will only increase the price for my calculations to 1.02$ and not to 1.20.
With the effect, that I “grow” into rising prices without ever risking selling too early again.
So today I finished my bot, it will start in 15 minutes to run, for a whole day long. Then we’ll see what happens.
Bot simulation
For my bot simulations I wanted to calculate as many possibilities as possible.
I have 3 triggers: sell low, rebuy lower, or rebuy higher (in case the price does not drop to the rebuy lower level). I wanted to see the outcome for all possibilities, in order for me to chose the right levels for the 3 triggers.
Let me give you an example to understand this:
Price of the asset is 10
It drops, sell trigger defines when to sell.
Is sell trigger at 10% then I would sell at 9.
If rebuy low trigger is another 10% lower, I would rebuy at 8 (for simplicity).
If however I am unlucky and the price starts rising after I sold in expectance the price would drop lower, then I surely don’t wanna miss out on the price increase either. So I have to, sour it is, bit the lemon, and rebuy at a higher level, say 10% too.
That means I sold at 9 and I will buy again if it drops to 8 (which is good) or rises to 11 (which is bad).
The same game I could make with 1%.
Meaning I sell at 9.9, rebuy at either 9.8 or 10.1.
You get the idea.
Now all those percentages, when the price changes by how much I sell, rebuy low or rebuy high:
there’s MANY possibilities.
And I want to make sure my bot uses one which proposes good enough profits.
I have now, started afresh 17th January, around 120k data points from Bitstamp, and around 400k data points from Binance.
So one simulation (eg. sell: -1%, low: -2%, high: 3%) will need to be calculated with 120k prices. Assumption, of course, is, that price development is always similar.
So what did I do?
I created my bot from the beginning, to be able to make simulation runs. I can feed as many simulations at once as I want. I have 3 levels, sell/low/high, I want to adjust.
My initial code looked like this for running this deep analysis was:
s = –0.3
down = –0.1
up = 0.1
count = 0
while s >= –4:
l = –2
while l >= –5:
h = –1
while h <= 5:
count += 1
h += up
print(“s:{}tl:{}th:{}”.format(s, l, h))
l += down
s += down
and the result something like this
s:-0.3 l:-2 h:5.099999999999998
s:-0.3 l:-2.1 h:5.099999999999998
s:-0.3 l:-2.2 h:5.099999999999998
s:-0.3 l:-2.3000000000000003 h:5.099999999999998
s:-0.3 l:-2.4000000000000004 h:5.099999999999998
s:-0.3 l:-2.5000000000000004 h:5.099999999999998
s:-0.3 l:-2.6000000000000005 h:5.099999999999998
s:-0.3 l:-2.7000000000000006 h:5.099999999999998
s:-0.3 l:-2.8000000000000007 h:5.099999999999998
s:-0.3 l:-2.900000000000001 h:5.099999999999998
s:-0.3 l:-3.000000000000001 h:5.099999999999998
s:-0.3 l:-3.100000000000001 h:5.099999999999998
s:-0.3 l:-3.200000000000001 h:5.099999999999998
s:-0.3 l:-3.300000000000001 h:5.099999999999998
s:-0.3 l:-3.4000000000000012 h:5.099999999999998
s:-0.3 l:-3.5000000000000013 h:5.099999999999998
s:-0.3 l:-3.6000000000000014 h:5.099999999999998
s:-0.3 l:-3.7000000000000015 h:5.099999999999998
s:-0.3 l:-3.8000000000000016 h:5.099999999999998
s:-0.3 l:-3.9000000000000017 h:5.099999999999998
s:-0.3 l:-4.000000000000002 h:5.099999999999998
s:-0.3 l:-4.100000000000001 h:5.099999999999998
s:-0.3 l:-4.200000000000001 h:5.099999999999998
s:-0.3 l:-4.300000000000001 h:5.099999999999998
my revised code is now.
s = –3
down = –1
up = 1
count = 0
while s >= –40:
l = –2
while l >= –50:
h = –1
while h <= 50:
count += 1
h += up
print(“s:{}tl:{}th:{}”.format(s, l, h))
l += down
s += down
and my revised result:
s:-3 l:-21 h:51
s:-3 l:-22 h:51
s:-3 l:-23 h:51
s:-3 l:-24 h:51
s:-3 l:-25 h:51
s:-3 l:-26 h:51
s:-3 l:-27 h:51
s:-3 l:-28 h:51
s:-3 l:-29 h:51
s:-3 l:-30 h:51
s:-3 l:-31 h:51
s:-3 l:-32 h:51
s:-3 l:-33 h:51
s:-3 l:-34 h:51
s:-3 l:-35 h:51
s:-3 l:-36 h:51
s:-3 l:-37 h:51
s:-3 l:-38 h:51
s:-3 l:-39 h:51
s:-3 l:-40 h:51
s:-3 l:-41 h:51
s:-3 l:-42 h:51
s:-3 l:-43 h:51
So now I have values I can just divide by 10 and they are as accurate as accurate can be with floating points.
Now this piece just made sure that the loops would create all the possibilities I wanted to calculate.
Obviously my bot logic and the log part (file or db) is missing.
That’s what I mean by decomplify your problems. Make one thing understood clearly, then add the next complexity level.
Profiling code
As it turns out for >100k price points my simulation run 2 hours. While, for 7 million decisions to take, quite acceptable, I wanted to improve on that.
Comes in cProfile
Add this to your code:
import cProfile
cProfile.run(‘main()’)
and it will give you a result like this:
613716 function calls in 0.530 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.529 0.529 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 classBitstampClient.py:18(__init__)
9999 0.260 0.000 0.523 0.000 classLogic.py:114(logic)
127159 0.037 0.000 0.037 0.000 classLogic.py:163(readyToSell)
391749 0.185 0.000 0.185 0.000 classLogic.py:176(readyToBuy)
7318 0.011 0.000 0.012 0.000 classLogic.py:273(simulateOrder)
9999 0.003 0.000 0.003 0.000 classLogic.py:303(getBalances)
9999 0.018 0.000 0.024 0.000 classLogic.py:347(updateStats)
1 0.000 0.000 0.000 0.000 classLogic.py:88(initLogic)
10027 0.002 0.000 0.002 0.000 classLogicPlayGround.py:102(logInfo)
1 0.000 0.000 0.000 0.000 classLogicPlayGround.py:105(__del__)
9999 0.001 0.000 0.001 0.000 classLogicPlayGround.py:108(updateHeartBeat)
1 0.000 0.000 0.000 0.000 classLogicPlayGround.py:16(__init__)
7370 0.001 0.000 0.001 0.000 classLogicPlayGround.py:99(logSimulation)
1 0.006 0.006 0.529 0.529 fetchAllAtOnce.py:40(main)
1 0.000 0.000 0.530 0.530 {built-in method builtins.exec}
9999 0.001 0.000 0.001 0.000 {built-in method builtins.len}
3 0.000 0.000 0.000 0.000 {built-in method builtins.print}
2 0.000 0.000 0.000 0.000 {built-in method now}
10051 0.002 0.000 0.002 0.000 {method ‘append’ of ‘list’ objects}
1 0.000 0.000 0.000 0.000 {method ‘commit’ of ‘sqlite3.Connection’ objects}
1 0.000 0.000 0.000 0.000 {method ‘disable’ of ‘_lsprof.Profiler’ objects}
53 0.000 0.000 0.000 0.000 {method ‘format’ of ‘str’ objects}
9979 0.003 0.000 0.003 0.000 {method ‘pop’ of ‘list’ objects}
1 0.000 0.000 0.000 0.000 {method ‘strftime’ of ‘datetime.date’ objects}
Run this over a decent amount of data points, not all, Ogure out where you can improve your data, improve it, test again, see the difference and stick with the change or revert back.
I could improve with 2 little changes almost 25% performance, which is 30 minutes on a 2 hour run.
But over time, seeing which ranges totally never make sense, I can fine-tune, eliminating some trigger bands for sell/low/high and thus increase simulation time even further.
Sending messages to running code
Up till now I didn’t know it’s possible to send a “message” to an already running piece of code. Okay I knew about event handlers, but not that “sending signals” (or messages) is possible.
So I figured my bot is doing normally quite well, but in times of very high volatility, or very steep price increases, he gets lost and I want to stop trading during these moments.
So far I’d have to kill the process, change the script which runs the process every hour, to not run the process. And if I wanted to start trading again I would undo those changes.
Now with signals it’s quite easy.
I send a signal, I use USR1/USR2 (for stop/start trading) and intercept and change a flag in my logic.
I have a flag: executingTrades. I simply switch this flag and my bot stops or restarts trading.
How to send a signal?
Use the kill command in a shell script:
#!/bin/zsh
# find the pid from the particular bot I want to stop trading
pid=`pgrep -f “[C]ellar.*websocket_temp.py”`
# send signal usr1
kill -s usr1 $pid
echo “websocket_temp.py trading stopped”
In the python code for the websocket you simply add these lines before going into .run_forever
signal.signal(signal.SIGINT, receiveSignal)
signal.signal(signal.SIGUSR1, receiveSignal)
signal.signal(signal.SIGUSR2, receiveSignal)
and the appropriate function which handles the signals
def receiveSignal(signalNumber, frame):
global logic
try:
if signalNumber == signal.SIGINT:
ticker_ws.close()
elif signalNumber == signal.SIGUSR1:
logic.simulate = False
elif signalNumber == signal.SIGUSR2:
logic.simulate = True
except Exception as e:
print(e)
Tried and tested on a bot which just consumes but does not act on the messages coming from the websocket.