IDX Screener — Engineering Handbook#
Everything a new engineer needs to understand, run, deploy and operate this platform. Written to be read top to bottom once, then used as a reference.
- Narrower documents: API.md (REST reference), FAIR-INTEGRATION.md (one client's flow)
- This document is the only one that covers the whole system.
1. What this is#
A stock screening platform for the Indonesia Stock Exchange (IDX). It ingests daily price data and broker transaction data, computes a large feature set over it, and runs several machine-learning and rule-based models that answer one question: which IDX stocks look interesting right now, and at what entry, stop and target?
A second family of models answers a different question: I am holding these positions and some are under water — what is the realistic recovery path?
The analysis is delivered through three independent surfaces, all reading the same data and the same model code:
| Surface | Who uses it | Entry point |
|---|---|---|
| Streamlit web app | Internal analysts, logged-in users | app.py |
| Telegram bot | Traders, conversational via Gemini | bot/telegram_bot.py |
| REST API | Third-party applications | api/server.py |
The platform is Indonesian-facing: user-visible copy in the bot and the Streamlit app is in Bahasa Indonesia. Code, comments and these docs are in English.
What it is not#
- Not a trading system. It produces analysis, never orders.
- Not real-time. The dataset is refreshed out of band, typically nightly, and the models score the most recent completed trading day.
- Not a data pipeline. The ingestion that produces the
.pkland.parquetfiles lives outside this repository (a Colab notebook writing to Google Drive). This repo consumes those artefacts.
2. Architecture#
┌──────────────── artefacts produced OUTSIDE this repo ───────────┐
│ idx_advanced_gross_model_multiindex.pkl (OHLCV + broksum) │
│ broker_detail.parquet (per-broker daily) │
│ recovery_model_v4.pkl (trained Model E) │
└────────────────────────────┬───────────────────────────────────-┘
│ read-only
┌──────────────────▼──────────────────┐
│ core/ — the shared engine │
│ │
│ data_provider.py load + cache │
│ feature_engine.py ~60 features │
│ screener_service.py headless runner │
│ usage_store.py unified ledger │
└───┬──────────────┬──────────────┬────┘
│ │ │
┌────────────────▼───┐ ┌───────▼────────┐ ┌──▼──────────────┐
│ app.py │ │ bot/ │ │ api/ │
│ Streamlit UI │ │ Telegram+Gemini│ │ FastAPI REST │
│ :8508 │ │ long-polling │ │ :8600 │
└────────────────────┘ └────────────────┘ └─────────────────┘
│ │ │
nginx + TLS Telegram API nginx + TLS
multiscreener.qomu.ai dev.multiscreener.qomu.ai
The important architectural property: the three surfaces share core/ and
screeners/ but never import each other. The bot does not import api, the API
does not import bot. Anything they both need lives in core/.
They are three separate OS processes under systemd, running from one directory with one virtualenv, reading one set of data files.
3. Repository layout#
app.py Streamlit entry point — auth gate, screener dropdown, layout
config/
settings.py ALL paths, env vars and shared constants. Start here.
users.yaml Streamlit logins (plaintext — see Known issues)
telegram_users.yaml Telegram allowlist, keyed by numeric user id
api_keys.yaml API keys (SHA-256 hashes; created by scripts/api_keys.py)
core/ Shared engine — imported by all three surfaces
screeners/ The models. Drop a file in, it auto-registers.
bot/ Telegram bot + Gemini conversational layer
api/ FastAPI REST service
scripts/ Operational CLIs (key management, usage, universe refresh)
tests/ pytest suite (ledger, attribution)
training/ Offline model training (produces recovery_model_v4.pkl)
docs/ This handbook, API docs, the served mockup
data/
processed/ Large artefacts: .pkl, .parquet (NOT in version control)
*.db SQLite stores, created on first write
deploy/ systemd unit + nginx site + install script (server only)
4. The data layer#
4.1 Main PKL — data/processed/idx_advanced_gross_model_multiindex.pkl#
The primary dataset. Daily OHLCV plus gross broker summary ("broksum") per ticker per day.
- Index: MultiIndex
(Ticker, Date) - Size: ~45 MB, ~543,000 rows, 962 tickers, history back to 2024-05-20
- Columns (22 total):
| Group | Columns |
|---|---|
| OHLCV | Open, High, Low, Close, Volume |
| Regular market | reg_value |
| Accumulation | accum_top3_val, accum_top5_val, dist_top5_val |
| Flow | bandar_net_flow, foreign_net_val, retail_net_val |
| Concentration | top1_dominance, top3_avg_buy_price |
| Negotiated market | nego_val, nego_avg_price, nego_premium |
| Derived signals | markup_signal, bandar_strength, smart_money_divergence |
"Bandar" is Indonesian market slang for a large operator moving a stock.
bandar_net_flow is the net value accumulated by the top brokers — the platform's
core edge signal.
Coverage characteristics worth knowing: on a healthy day roughly 921 of 962
tickers have a Close, ~830 have non-zero reg_value (i.e. actually traded), and
~105 have negotiated-market activity. Tickers with no trading still appear as rows
with zeros — absence of a row is not how "didn't trade" is represented.
4.2 Broker detail — data/processed/broker_detail.parquet#
Per-broker, per-ticker, per-day transaction detail. This is the granular source the broksum columns above are aggregated from.
- Size: ~286 MB, 21.3 million rows, history back to 2024-01-29
- Columns:
Date,Ticker,Market,Broker,Action,Type,Is_Foreign,Is_Retail,Gross_Lots,Gross_Value,Avg_Price MarketisREG(regular) orNEGO(negotiated). A typical day is ~38,000 rows across ~835 tickers, of which only ~350 are NEGO.
Read it with pyarrow filters, never fully — core/data_provider.py:load_broker_detail
pushes ticker/date/market predicates down to the file.
4.3 Recovery model — data/processed/recovery_model_v4.pkl#
This is a trained model, not data. ~991 MB. Loaded by Model E (recovery_v4)
to predict price trajectories. Unlike the other models, which retrain on every
run, Model E loads this pre-trained bundle.
Regenerate with training/train_recovery.py. Note that its age matters: a bundle
trained months ago scores today's prices with months-old parameters.
4.4 SQLite stores — data/*.db#
Five stores, each owned by one module. All created lazily on first write.
| File | Owner | Tables |
|---|---|---|
usage.db |
core/usage_store.py |
usage_events, consents |
user_state.db |
bot/user_state_store.py |
watchlist, portfolio, alerts, user_prefs, signal_alert_log |
chat_history.db |
bot/chat_store.py |
messages |
feedback.db |
bot/feedback_store.py |
feedback |
broadcast.db |
bot/broadcast_store.py |
broadcast_picks |
House pattern for all of them: flat module functions, no ORM, connection
opened and closed per call, CREATE TABLE IF NOT EXISTS on connect as the
migration strategy, every function wrapped in try/except returning a benign
default.
usage.db is the exception and deliberately so — it is the only store written by
two processes (the API and the bot), so it uses WAL, a busy timeout, and
PRAGMA user_version migrations. See §9.
4.5 Config files#
| File | Purpose | Secret? |
|---|---|---|
config/settings.py |
Paths, env var wiring, shared constants | no |
config/users.yaml |
Streamlit logins | yes — plaintext passwords |
config/telegram_users.yaml |
Telegram allowlist, int user id → {username, role} |
identities |
config/api_keys.yaml |
API keys as SHA-256 hashes, mode 600 | hashes only |
.env |
Secrets + runtime flags, loaded by systemd EnvironmentFile |
yes |
Nothing in the code loads .env. It only takes effect because the systemd
units declare EnvironmentFile=. Running a process by hand without exporting
those variables gives you defaults, not production behaviour — a real source of
confusion when testing.
5. The feature engine#
core/feature_engine.py turns the 22 raw columns into roughly 60 features
that the models consume. run_base_feature_pipeline(df) is the single entry
point; everything else is a step inside it.
| Family | Function | Produces |
|---|---|---|
| Volatility | add_atr |
ATR_14 |
| Momentum | add_rsi |
rsi_14 |
| Bands | add_bollinger |
ma20, std_20, upper_band, lower_band, bb_width |
| Returns | add_daily_returns |
daily_ret, rel_strength |
| Liquidity | add_volume_features |
AvgValue_20d, AvgVolume_20d, vol_ratio, rvol, vol_dry_up |
| Broker flow | add_bandar_features |
Intensity, Accum_Intensity, Markup_Smooth, Smart_Money_Ratio, Bandar_Flow_5d, Bandar_Accum_2W |
| Regime | calculate_market_regime |
regime_score, mkt_volatility |
| Interactions | add_interaction_features |
Flow_Divergence, Accum_Squeeze, Whale_Momentum |
| Risk | add_overextension_risk |
dist_ma20_norm, Overextension_Risk |
| Crossovers | add_golden_cross_features |
gc_ma5_ma20, gc_macd_line, gc_stoch_kd plus _days and _spread variants |
Two things to know:
regime_scoreis market-wide, not per-stock: a single number per day describing whether IDX is bullish, sideways or bearish.get_regime_label()andget_regime_emoji()turn it into display text. Several models branch on it.sanitize_for_xgboostexists because XGBoost rejects infinities and some NaN patterns. Every model calls it before training. If you add a feature that can divide by zero, it is handled here or it breaks training.
Computed features are cached in-process for one hour
(screener_service._get_featured_data, _FEATURE_CACHE_TTL = 3600). The first
call after a restart pays the full pipeline cost — measured at 30–45 seconds.
6. The models#
Models live in screeners/. Auto-discovery: screeners/__init__.py scans the
directory on import, imports every .py, and registers any BaseScreener
subclass by its metadata().id. Adding a model is dropping in a file — there is
no registry to edit.
6.0 The contract#
class BaseScreener(ABC):
@staticmethod
def metadata() -> ScreenerMetadata # id, name, description, category, version
def render_sidebar(self) -> dict # Streamlit widgets → params
def run(self, params: dict) -> Any # the actual analysis
def render_results(self, results, params) # Streamlit display
run() is the only method the bot and API call. The other two are Streamlit-only,
which is why headless callers must install the Streamlit stubs first (§7.1).
Eleven models register; six are exposed over the API. The rule is mechanical:
a model appears in the API if and only if it has an entry in
screener_service.PARAM_SCHEMAS, because without a schema there is no
validatable contract.
| id | Name | Category | API | File |
|---|---|---|---|---|
regime_adaptive_v6 |
Model A: Regime Adaptive | swing | ✅ | regime_adaptive_v6.py |
gann_classic_v7 |
Model B: Cycle Analysis | swing | ✅ | gann_classic_v7.py |
gross_alpha_v10 |
Model C: General Purpose | swing | ✅ | gross_alpha_v10.py |
confluence_filter |
Model D: Confluence Filter | swing | ✅ | confluence_filter.py |
recovery_v4 |
Model E: Portfolio Recovery | recovery | ✅ | recovery_v4.py |
model_f_combined |
Model F: Combined ML + Signals | swing | ✅ | model_f_combined.py |
gann_planner_v9 |
Model G: Dynamic Cycle-AI Planner | position | ✖ | gann_planner_v9.py |
backtest_lab |
Backtest Lab | experimental | ✖ | backtest_runner.py |
confluence_backtest |
Backtest D | experimental | ✖ | confluence_backtest.py |
model_f_backtest |
Backtest F | experimental | ✖ | model_f_backtest.py |
recovery_backtest |
Backtest E | experimental | ✖ | recovery_backtest.py |
Note backtest_lab lives in backtest_runner.py — the only model whose id does
not match its filename.
6.1 The shared swing pipeline#
Models A, B, C, F and G are variations on one pipeline. Understand this once and each model is then just its differences.
- Load the cached feature frame (
_get_featured_data, §5). - Filter liquidity —
min_price,min_val/min_daily_val,min_voldrop names too thin to trade. Typically ~830 of 962 tickers survive. - Apply the FCA filter —
apply_fca_filter()removes stocks under IDX's Notasi Khusus special-notation watchlist. - Label history with
core/backtester.create_order_aware_labels(). For each historical row: withinhold_days, did price hit+target_gainbefore−stop_loss? The "order-aware" part matters — a naive implementation that checks max-gain and max-loss independently will label a trade a winner even when the stop was hit first. The docstring calls this out as a fixed temporal ordering bug. - Train an XGBoost classifier on the labelled history, excluding the analysis date. This happens on every single run — there is no model cache. It is the reason a run costs 9–16 s and why the API is asynchronous.
- Score the analysis date →
Probabilityper ticker (0–100). - Filter and rank by the model's own strategy, cut to
max_picks. - Size positions — most models compute suggested allocation from
capital,max_alloc,risk_capand the measured win rate/payoff.
Each model also returns win_rate and payoff from its own test split, which is
the honest way to read its picks: a 70% probability from a model with a 0.15 win
rate means something very different from the same number at 0.55.
6.2 Model A — regime_adaptive_v6 (Regime Adaptive, v6.1)#
The idea: a pattern that works in a bull market does not work in a bear market, so train only on history that looks like today.
What makes it different. Where other models train on all available history,
Model A computes the current market regime_score and then selects training rows
whose own regime_score falls within ±0.20 of it:
tol = 0.20
regime_mask = (valid['regime_score'] >= current_regime - tol) & \
(valid['regime_score'] <= current_regime + tol)
The classifier therefore learns "what worked in conditions like these" rather than a blended average across every regime. The trade-off is a smaller training set — in a rare regime there may be little matching history, and the model is correspondingly less confident.
It also adds two interaction features the others do not use: Accum_Squeeze
(accumulation happening while volatility compresses) and Whale_Momentum (large
broker flow aligned with price momentum).
Key parameters: capital, min_price, min_val, min_vol, target_gain
(0.10), stop_loss (0.05), hold_days (5), max_alloc, min_intensity,
min_probability.
Returns: df_result, diag, win_rate, payoff, train_positives,
train_total. Results include the regime label so the UI can show the market
state the picks were produced under.
When it suits: trending markets with a clear direction. The model's own
guidance (core/model_docs.py) tells users that results are best when
regime_score is above +0.20 or below −0.20, and to consider waiting when the
market is sideways.
6.3 Model B — gann_classic_v7 (Cycle Analysis, v7.7)#
The idea: price moves in cycles off pivot lows, and both the distance from a pivot and the time elapsed since it carry information.
What makes it different. It layers Gann-style cycle features on top of the shared pipeline before training:
| Feature | Meaning |
|---|---|
Days_Since_Low / Days_Since_High |
Bars since the last 60-day pivot |
Cycle_PnL |
Gain from the pivot low — where in its run the stock is |
Gann_Angle_Support |
Pivot low + Days_Since_Low × ATR/2 — a rising support line |
Dist_Gann_Angle |
How far price sits above/below that line |
Gann_Res_360 |
(√pivot_low + 1)² — the classic square-of-nine resistance |
Low_Cycle_90 / High_Cycle_90 |
Position within a 90-day cycle, as a modulo |
Pivots are detected as rolling 60-day extremes, then cumsum groups the bars
belonging to each cycle so "days since" is computed per cycle rather than globally.
Key parameters: adds enable_gann (bool), gann_mode (Bullish/Bearish)
and lookahead to the standard set. Defaults are wider than the other swing
models — target_gain 0.15, stop_loss 0.08, hold_days 20 — because cycle
setups are expected to take longer to play out.
When it suits: longer holds, and stocks with genuinely cyclical behaviour (commodities, cyclical blue chips). Its own documentation says to skip it in choppy markets, where cycles are not well defined.
6.4 Model C — gross_alpha_v10 (General Purpose, v10.0)#
The idea: the broker summary — who is accumulating, how concentrated it is, whether it diverges from price — is the edge. This is the platform's "bandarmology" model and its general-purpose default.
What makes it different. It adds a volume/volatility feature block on top of the base pipeline:
| Feature | Meaning |
|---|---|
avg_price_{3,5,10,20}d, avg_vol_{…}d |
Multi-period trend filters |
rvol |
Volume relative to its 20-day mean |
vol_dry_up |
20-day mean ÷ 60-day mean — volume drying up before a move |
Volatility_20d |
Rolling return volatility |
base_tightness |
1 / volatility — a tight base scores high |
Combined with the base broker features (bandar_net_flow, Accum_Intensity,
Smart_Money_Ratio, Flow_Divergence), the model is looking for quiet
accumulation in a tight base rather than momentum that has already happened.
_generate_context() turns each pick's feature values into a human-readable
sentence, which is what the Telegram bot shows alongside a pick.
Key parameters: capital, max_alloc (0.25), risk_cap (0.05),
target_gain (0.10), stop_loss (0.05), hold_days (5), min_intensity
(0.05), min_price (200), min_daily_val (1e9), min_daily_vol,
vol_price_days, filter_mode, retrain, min_probability.
A note on min_intensity: this one filter is frequently why Model C returns
nothing. On a small or quiet universe it can eliminate every candidate before
probability is even considered — the funnel will show after_intensity: 0.
Loosening it is usually the right first move, and the empty-results explainer
says so.
When it suits: the default choice for 5-day swings. It is one of the two models the daily broadcast runs.
6.5 Model D — confluence_filter (Confluence Filter, v1.0)#
The idea: no machine learning at all. Look for stocks where several classic technical signals fire together or in close sequence.
The sequence it hunts for, in the order these normally occur:
- Stochastic %K crosses up out of oversold — earliest, catches the reversal
- MACD histogram flips positive — momentum shifting
- MACD line crosses above signal — momentum confirmed
- MA5 crosses above MA20 — trend confirmed
_detect_signals() computes each as a boolean per bar; _count_recent_signals()
counts how many fired within the lookback window; stocks with at least
min_signals pass, with volume above average as confirmation.
Because there is no training step it is the fastest model — a few seconds rather than fifteen — and it is fully deterministic: the same inputs always give the same picks.
Key parameters: lookback (5), min_signals (2), stoch_period,
stoch_oversold, macd_fast/slow/signal, ma_fast/ma_slow, min_vol_ratio,
and use_stoch / use_hist / use_macd / use_ma to enable each signal
independently.
Returns: df_result, diag, total_scanned, total_hits,
excluded_seller, signal_distribution — the last being a histogram of how many
stocks had 0, 1, 2, 3 signals, which is useful for calibrating min_signals.
When it suits: when you want an explainable, auditable screen with no model risk. It is the second model the daily broadcast runs.
6.6 Model E — recovery_v4 (Portfolio Recovery, v4.1)#
Completely different from every other model. It does not screen the market — it analyses positions you already hold, most of which are losing, and answers "what is the realistic recovery path?"
It does not train. It loads the pre-trained bundle
data/processed/recovery_model_v4.pkl (~991 MB) and runs inference. This is why
it is both the fastest model in warm steady state and the only one that will not
run at all without its artefact.
What it predicts, per position, at a horizon of 10/20/30/40 trading days:
- A trajectory distribution over five classes:
CRASH / DECLINE / SIDEWAYS / CLIMB / RALLY - Directional probabilities —
p_up,p_down,p_neutral - A return quantile grid — p10/p25/p50/p75/p90 at several horizons
- Support and resistance from
_compute_sr()over a 60-day lookback
make_decision() turns that into advice. Its central calculation is the
breakeven requirement: a position down 41.8% needs 0.418/(1−0.418) = +71.8%
to get back to cost. The function compares that requirement against the model's
own p75 upside and produces a verdict — the reason strings users see, such as
"Recovery unlikely: need +72%, P75 only +8%". It also decides an
optimal_exit_days window and a plan (hold / average down / cut).
Input shape — each position takes either form, not both:
{"ticker": "GOTO", "pnl_input": -41.81} // percent
{"ticker": "GOTO", "avg_price_input": 96.24} // your average cost
Returns: positions, horizon, analysis_date. Each position carries
outlook (BULLISH/NEUTRAL/BEARISH), reason, plan (target, stop_loss,
exit_window, and a large explanation dict) and prediction (trajectory,
probabilities, quantiles).
Two sharp edges:
- An empty
positionsarray fails the run — there is nothing to analyse. The error is"⚠️ No positions entered." - It uses lowercase
tickerin its output while every other model usesTicker.
Naming inconsistency to be aware of: recovery_v4.py:27 defines
TRAJ_CLASSES with CLIMB, but config/settings.py:123 still lists the older
BOUNCE_FADE. Line 223 handles both for backward compatibility with older PKLs,
so nothing breaks — but settings.py is the stale one, and CLIMB is what the
API actually returns.
6.7 Model F — model_f_combined (Combined ML + Signals, v1.0)#
The idea: Model C knows probability, Model D knows confluence. Combine them.
Model F computes both — v10 features and Model D's signal detection (it
imports _detect_signals and _count_recent_signals directly from
confluence_filter) — then applies one of four selection strategies:
strategy |
Logic |
|---|---|
Gated |
Filter to confluence_count ≥ min_signals, then rank by probability. Quality over quantity |
Hybrid Score |
Blended score: prob_weight × prob + (1−prob_weight) × signals, keeping anything that passes probability or has ≥1 signal |
Intersection |
Strict: must pass both min_prob and min_signals. Fewest picks, highest conviction |
Inverse Signal |
The default and the backtest winner. score = prob_weight × (prob/100) − signal_penalty × (signals/4) |
Compare All |
Runs all four and returns them side by side, plus an overlap count |
Inverse Signal deserves explanation because it looks wrong at first glance:
it penalises confluence. The reasoning is that by the time four signals have
fired, the move is visible to everyone and you are late. It prefers high model
probability with few visible signals — accumulation the market has not yet
priced. Backtesting made it the default.
Compare All is genuinely useful during analysis: a stock appearing in all four
strategies is a consensus pick.
Key parameters: strategy, prob_weight (0.6, range 0.3–0.8),
signal_penalty (0.3, range 0.0–0.5, only affects Inverse Signal),
min_signals, min_probability, lookback (2 — fresh signals only),
target_gain, stop_loss, hold_days, max_picks, min_price (500),
min_daily_val (1e10 — notably stricter than Model C).
Returns: picks (not df_result), target_date, win_rate, payoff,
n_candidates, diag. Under Compare All it also returns gated, hybrid,
intersection, inverse_signal and overlap.
Caveat in the code's own words: it "does NOT broadcast to the daily channel — runs only in Streamlit for exploration." It is nonetheless the model the FAIR mobile client uses, via the API.
6.8 Model G — gann_planner_v9 (Dynamic Cycle-AI Planner, v9.4)#
Not exposed over the API — it has no PARAM_SCHEMAS entry. Reachable from
Streamlit only. Adding a schema would publish it.
The most elaborate model in the repo. On top of Gann cycle features it adds:
- Walk-forward validated training (
_wf_train) rather than a single split - Composite flow scoring (
_calc_flow_score) blending flow trend, relative flow, cycle P&L and days-since-low, including shakeout detection - Fast regime detection (
_fast_regime) with four states —BULL / BULL_WEAKENING / CHOPPY / BEAR— finer than the singleregime_score - Market stress detection from 5-day returns, breadth and regime delta
- Pairwise correlation deduplication so picks are not five names that move together
- Probability-weighted position sizing and regime-adaptive target capping
Category position rather than swing: it is built for longer holds.
6.9 The backtest models#
Four experimental entries, all excluded from the API. They exist to validate the live models rather than to produce picks.
| id | What it validates |
|---|---|
backtest_lab |
Walk-forward harness for the ML models generally (backtest_runner.py) |
confluence_backtest |
Model D: recomputes signals at each historical eval date, simulates the resulting trades. No training involved |
model_f_backtest |
Model F: a custom walk-forward loop that computes v10 features and D signals once, then trains at each eval date and compares all strategies |
recovery_backtest |
Model E — and this one is not a trade backtest. Model E predicts trajectory classes and outlook, not buy/sell signals, so this audits whether those predictions correlate with realised outcomes |
The shared engine is core/backtester.py (661 lines), which also provides
create_order_aware_labels() used by every live ML model.
6.10 Result shapes are not uniform#
This catches people, so it is worth stating plainly:
| Model | Picks key | Ticker key | Date key |
|---|---|---|---|
| A, B, C, D | df_result |
Ticker |
target_date / data_date |
| F | picks |
Ticker |
target_date |
| E | positions |
ticker (lowercase) |
analysis_date |
core/usage_store.summarize_result() and api/views.py both probe all three
defensively — copy that approach rather than assuming one.
Raw results also mix display strings with numbers. df_result rows carry
"Last Close": "8,050" and "Daily Val": "139.4B" next to genuine floats,
because they are rendered straight into Streamlit tables. The API's
view=compact exists to give clients real numbers instead.
7. Surface 1 — the Streamlit app#
app.py, served on 127.0.0.1:8508, published at multiscreener.qomu.ai.
Flow: core/auth.py gates on config/users.yaml → a sidebar dropdown built from
the screener registry → the selected model's render_sidebar() collects params →
run() → render_results().
Notes:
- Login is a plaintext password compare. See Known issues.
- core/model_docs.py supplies the plain-language "ℹ️ Tentang model ini" text.
- core/empty_results_explainer.py renders the "nothing passed your filters"
panel with suggestions for which filter to loosen. The API reuses its
suggestion logic for the same purpose.
- Streamlit builds params through each screener's own render_sidebar() and
never touches PARAM_SCHEMAS — so API schema changes cannot affect the UI.
7.1 Running screeners headlessly#
screeners/*.py call st.toast, st.spinner, st.session_state and friends
directly. To run them outside Streamlit,
core/screener_service._install_streamlit_stubs() monkey-patches those into
no-ops and turns @st.cache_data into a real in-memory cache.
Every headless entry point must call it before importing screeners. The bot,
the API and every script in scripts/ do.
8. Surface 2 — the Telegram bot#
bot/telegram_bot.py, long-polling, no webhook.
Identity#
The canonical id is the integer Telegram user id. config/telegram_users.yaml
maps it to {username, role} with roles user < pro < admin. The file is
hot-reloaded on mtime, so adding a user takes effect without a restart. If the
file is empty the bot runs in open mode and allows everyone — worth knowing
before you clear it.
Users added by @handle get a pending_<username> placeholder key, upgraded to
their real integer id on first message by _backfill_user_id.
Conversational layer#
bot/gemini_parser.py is the brain. A user message goes to Gemini with a set of
declared tools; Gemini decides which to call; _execute_tool_call dispatches to
the service layer. Tools cover running screeners, quotes, broker flow, nego
activity, turning points, watchlist and portfolio management, and feedback.
Without a GEMINI_API_KEY it degrades to _fallback_chat, a keyword/regex parser
that can still run screeners.
Scheduled jobs#
| Job | Time (WIB) | What |
|---|---|---|
daily_broadcast |
08:00 | Runs Models C and D, posts top picks to the channel |
user_daily |
07:30 | Per-user DMs: turning-point alerts, price alerts, digest |
usage_prune |
03:30 | Trims the usage ledger past its retention window |
Both pushes are gated by PUSH_ENABLED. Set PUSH_ENABLED=0 in .env and
neither is scheduled at all — the scheduler registers zero push jobs. Admin
commands /broadcast, /broadcasttest and /digest are deliberate manual
actions and are not gated.
Commands#
30 registered. User-facing: /start /help /list /clear /watch /unwatch /watchlist
/port /unport /portfolio /alert /alerts /delalert /signals /whoami.
Admin-only, DM-only: /adduser /adduserid /removeuser /users /confirm
/exportfeedback /feedbackstats /usage /chathistory /broadcast /broadcasttest
/digest /pick /winrate /winrateat /dm.
9. Surface 3 — the REST API#
api/server.py, uvicorn on 127.0.0.1:8600, published at
dev.multiscreener.qomu.ai. Full reference in API.md.
Layout#
Everything machine-facing is under /api; human-facing pages sit at the root.
/api/v1/models /api/v1/jobs /api/v1/data
/api/v1/usage /api/v1/consent /api/v1/admin/usage
/api/health /api/docs /api/openapi.json
/guide /guide/integration /guide/reference /mockup
Module map#
| File | Role |
|---|---|
server.py |
App, middleware, host allowlist, legacy redirects, system routes |
auth.py |
API keys, scopes, rate limiting, X-End-User validation |
jobs.py |
Background run execution, two-worker pool, TTL eviction |
routes_models.py |
Catalogue, schemas, run submission, the analysis-date pin |
routes_jobs.py |
Poll jobs; two-tier live/ledger contract |
routes_data.py |
14 market-data endpoints |
routes_usage.py |
Tenant usage + separate admin router |
routes_consent.py |
Disclaimer gate |
views.py |
view=compact card shapes |
validation.py |
Rejects unknown/out-of-range params |
serialization.py |
numpy/NaN/Timestamp → strict JSON |
errors.py |
One error envelope + request ids |
docs_pages.py |
Renders docs/*.md at /guide/* |
Two identities#
Authorization: Bearer idxs_live_… ← the consuming APPLICATION
X-End-User: nasabah_00471 ← the end CUSTOMER (required on runs)
Keys are random 256-bit secrets stored as SHA-256 digests — not bcrypt, because
high-entropy secrets cannot be dictionary-attacked and the API hashes on every
request. Scopes: models:read, models:run, data:read, usage:read,
usage:admin. The key file is hot-reloaded on mtime, so issuing and revoking
take effect within a second with no restart.
Why runs are asynchronous#
A run retrains the model: 9–16 s warm, 30–45 s cold. POST .../runs returns
202 with a job id to poll, or 200 inline if ?wait=N (max 60) was satisfied.
Concurrency is capped at 2 workers because each run pins a large DataFrame and
trains a model — more workers make every run slower, not faster.
The usage ledger#
core/usage_store.py records every model run from all three channels into one
table, so "who ran what, with which parameters" is a single query. Written by two
processes, hence WAL, a 5 s busy timeout, and PRAGMA user_version migrations.
Writes are best-effort and fail silently by design — a ledger problem must
never break a model run. That makes GET /api/health → usage.writable the only
way to notice it has stopped working. Reads deliberately do raise: losing a
write is acceptable, lying about history is not.
10. Running it locally#
python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt
# Data artefacts are NOT in version control. Either copy the real ones into
# data/processed/, or use the sample bundle (see export/README.md).
# Streamlit
.venv/bin/streamlit run app.py
# REST API
.venv/bin/python scripts/api_keys.py create local-dev \
--scopes models:read,models:run,data:read
.venv/bin/uvicorn api.server:app --host 127.0.0.1 --port 8600
# Telegram bot (needs TELEGRAM_BOT_TOKEN, and GEMINI_API_KEY for conversation)
export TELEGRAM_BOT_TOKEN=... GEMINI_API_KEY=...
.venv/bin/python -m bot.telegram_bot
# Tests
.venv/bin/python -m pytest tests/ -q
tests/test_usage_store.py runs in under a second because core/usage_store.py
is import-light by design. tests/test_attribution.py takes ~30 s because
importing the bot pulls in the Gemini and Telegram clients.
Environment variables#
| Variable | Default | Effect |
|---|---|---|
TELEGRAM_BOT_TOKEN |
— | Bot auth. Without it the bot will not start |
GEMINI_API_KEY |
— | Conversational layer; absent → regex fallback |
PUSH_ENABLED |
1 |
0 disables both scheduled pushes |
BROADCAST_CHANNEL_ID |
a channel id | Comma-separated targets; empty also disables the broadcast |
BOT_MIN_PROBABILITY |
65 |
Default probability floor for bot picks |
IDX_MAIN_PKL / IDX_RECOVERY_PKL / IDX_BROKER_DETAIL |
filenames | Override which artefact is loaded |
IDX_USAGE_DB |
data/usage.db |
Ledger location |
IDX_USAGE_RETENTION_DAYS |
180 |
0 = keep forever |
IDX_API_KEYS_PATH |
config/api_keys.yaml |
Key file |
IDX_API_JOB_WORKERS |
2 |
Concurrent model runs |
IDX_API_ALLOWED_HOSTS |
(empty) | Comma-separated Host allowlist; empty = no restriction |
IDX_API_PIN_ANALYSIS_DATE |
(empty) | Clamp every run to this date — integration environments |
IDX_API_CORS_ORIGINS |
(empty) | CORS origins; empty = off |
11. Deployment#
The server#
One Ubuntu box, 103.150.100.165, hostname idx-experiment. Everything lives in
/home/prasma/multiscreener with a single .venv. Three systemd units, all
running as user prasma:
| Unit | Binds | Published as |
|---|---|---|
multiscreener-streamlit |
127.0.0.1:8508 |
multiscreener.qomu.ai |
multiscreener-telegram |
— (outbound polling) | — |
multiscreener-api |
127.0.0.1:8600 |
dev.multiscreener.qomu.ai |
Each unit uses EnvironmentFile=/home/prasma/multiscreener/.env and
Restart=always. Nothing binds a public interface — nginx terminates TLS and
proxies to localhost. An API key over plain HTTP is a key in the clear.
Deploying a change#
scp api/server.py prasma@multiscreener.qomu.ai:/home/prasma/multiscreener/api/
sudo systemctl restart multiscreener-api
Python loads modules once at import. Changing a file on disk does nothing to a
running process — a restart is always required, and until you do it the on-disk
and in-memory code differ. With Restart=always an unattended crash will pick up
the new code at an arbitrary hour, so restart deliberately rather than leaving
the gap open.
Never overwrite: config/*.yaml, .env, data/. Production accumulates users
and state there.
Standing up the API from scratch#
deploy/install-api.sh (run with sudo) installs the systemd unit, verifies
/api/health locally, installs the nginx site, runs nginx -t before every
reload so a mistake cannot take down the other vhosts, then issues the
certificate with certbot --nginx --redirect. Idempotent.
Sudo scope#
Passwordless sudo on this box is narrow:
(ALL) NOPASSWD: /bin/systemctl restart multiscreener-*,
/bin/systemctl status multiscreener-*,
/bin/journalctl -u multiscreener-*
Service restarts need no password; nginx, certbot, writing systemd units and
reading /etc all do. This is why services are named multiscreener-* — the
prefix is what makes them restartable unattended.
12. Operations#
Health#
curl -s https://dev.multiscreener.qomu.ai/api/health | jq
# data_loaded false right after a restart → first request pays the cold start
# usage.writable false means ledger writes are failing SILENTLY
# jobs.by_status queued piling up means runs are backing up
sudo systemctl status multiscreener-{api,telegram,streamlit}
sudo journalctl -u multiscreener-api -n 100 --no-pager
Common tasks#
# API keys
python scripts/api_keys.py create acme --scopes models:read,models:run
python scripts/api_keys.py list
python scripts/api_keys.py disable acme # effective within 1s, no restart
# Usage
python scripts/usage.py stats
python scripts/usage.py top --by model --days 30
python scripts/usage.py recent --channel telegram
python scripts/usage.py prune --days 180 --yes
# Refresh the ticker universe (manual: IDX sits behind Cloudflare)
python scripts/refresh_idx_universe.py --manual-tickers GetSecuritiesStock.json
Turning pushes off#
sed -i 's/^PUSH_ENABLED=.*/PUSH_ENABLED=0/' .env
sudo systemctl restart multiscreener-telegram
# verify: the scheduler should log ZERO "Added job" lines for pushes
journalctl -u multiscreener-telegram --since "-2 min" | grep "Added job"
Data freshness#
Every result carries the trading date it reflects (as_of, target_date,
analysis_date). Check staleness with:
import pandas as pd
from config.settings import Settings
df = pd.read_pickle(Settings.MAIN_PKL_PATH)
print(pd.to_datetime(df.index.get_level_values('Date')).max())
A healthy day has ~921 priced tickers and ~830 with non-zero reg_value. If
reg_value > 0 drops far below that, the scrape was partial even though the
file exists and the date looks current.
13. Known issues and sharp edges#
Ordered by how likely they are to hurt.
-
Plaintext passwords in
config/users.yaml. Streamlit login compares with==. The module docstring has said "hash with bcrypt" since the beginning. Anyone who reads that file has every account. -
The bot token is logged in plaintext.
httpxlogs request URLs at INFO and the token sits in the path, so everygetUpdatesline leaks it to anyone who can read the journal. One-line fix:logging.getLogger("httpx").setLevel(logging.WARNING). -
Memory headroom is thin. 8 GB total, no swap. The API alone held ~1.6 GB and the bot ~0.9 GB before either loaded the 991 MB recovery bundle. Two concurrent Model E runs could invoke the OOM killer, which will not necessarily pick the process that caused it. Mitigations: add swap, set
MemoryMaxon the API unit, or dropIDX_API_JOB_WORKERSto 1. -
The ingestion pipeline is external and unreliable. It lives in a Colab notebook outside this repo. Observed failures: 2026-09-07 negotiated-market data almost entirely missing; 2026-09-16 broker summary covering 206 of ~835 tickers; 2026-09-17 onward no run at all. Nothing in this repo monitors it. A partial day looks completely normal from the file system.
-
dev.multiscreener.qomu.aiis not a dev environment. It shares the directory, the virtualenv, the data files and the box with production. The name promises isolation that does not exist. -
Model E's bundle ages silently. It is loaded, not retrained, so its predictions quietly reflect whenever it was last trained while the prices it scores are current. Nothing surfaces that age.
-
No schema versioning except in
usage_store. The other four SQLite stores useCREATE TABLE IF NOT EXISTS, which silently no-ops against an older table. Adding a column there needs a manualALTER. -
nginxrouting depends on filename order. No block declaresdefault_serverfor 443, so an unmatched hostname lands on whichever site loaded first. The API'sIDX_API_ALLOWED_HOSTSguards against this from the application side; the nginx-level fix is still outstanding.
14. Glossary#
| Term | Meaning |
|---|---|
| IDX | Indonesia Stock Exchange |
| broksum | Broker summary — aggregated per-broker buy/sell per ticker per day |
| bandar | Market slang for a large operator moving a stock; bandar_net_flow tracks it |
| nego | Negotiated market — off-order-book block trades, distinct from REG |
| FCA | Notasi Khusus — IDX's special-notation watchlist of flagged stocks |
| confluence | How many independent technical signals agree on a stock |
| regime | Market-wide state: bullish / sideways / bearish, as regime_score |
| nasabah | Indonesian for "customer" — used as the end-user id in API examples |
| WIB | Western Indonesian Time, UTC+7 — all schedules are in WIB |
docs/HANDBOOK.md in the idx-screener repository. This page is rendered from that file, so the repo stays the single source of truth.