IDX Screener API · dev.multiscreener.qomu.ai

IDX Screener — Engineering Handbook#

Everything a new engineer needs to understand, run, deploy and operate this platform. Written to be read top to bottom once, then used as a reference.


1. What this is#

A stock screening platform for the Indonesia Stock Exchange (IDX). It ingests daily price data and broker transaction data, computes a large feature set over it, and runs several machine-learning and rule-based models that answer one question: which IDX stocks look interesting right now, and at what entry, stop and target?

A second family of models answers a different question: I am holding these positions and some are under water — what is the realistic recovery path?

The analysis is delivered through three independent surfaces, all reading the same data and the same model code:

Surface Who uses it Entry point
Streamlit web app Internal analysts, logged-in users app.py
Telegram bot Traders, conversational via Gemini bot/telegram_bot.py
REST API Third-party applications api/server.py

The platform is Indonesian-facing: user-visible copy in the bot and the Streamlit app is in Bahasa Indonesia. Code, comments and these docs are in English.

What it is not#


2. Architecture#

            ┌──────────────── artefacts produced OUTSIDE this repo ───────────┐
            │  idx_advanced_gross_model_multiindex.pkl   (OHLCV + broksum)    │
            │  broker_detail.parquet                     (per-broker daily)   │
            │  recovery_model_v4.pkl                     (trained Model E)    │
            └────────────────────────────┬───────────────────────────────────-┘
                                         │ read-only
                      ┌──────────────────▼──────────────────┐
                      │  core/  — the shared engine          │
                      │                                      │
                      │  data_provider.py   load + cache     │
                      │  feature_engine.py  ~60 features     │
                      │  screener_service.py headless runner │
                      │  usage_store.py     unified ledger   │
                      └───┬──────────────┬──────────────┬────┘
                          │              │              │
         ┌────────────────▼───┐  ┌───────▼────────┐  ┌──▼──────────────┐
         │ app.py             │  │ bot/           │  │ api/            │
         │ Streamlit UI       │  │ Telegram+Gemini│  │ FastAPI REST    │
         │ :8508              │  │ long-polling   │  │ :8600           │
         └────────────────────┘  └────────────────┘  └─────────────────┘
                  │                      │                    │
            nginx + TLS            Telegram API        nginx + TLS
         multiscreener.qomu.ai                     dev.multiscreener.qomu.ai

The important architectural property: the three surfaces share core/ and screeners/ but never import each other. The bot does not import api, the API does not import bot. Anything they both need lives in core/.

They are three separate OS processes under systemd, running from one directory with one virtualenv, reading one set of data files.


3. Repository layout#

app.py                  Streamlit entry point — auth gate, screener dropdown, layout
config/
  settings.py           ALL paths, env vars and shared constants. Start here.
  users.yaml            Streamlit logins (plaintext — see Known issues)
  telegram_users.yaml   Telegram allowlist, keyed by numeric user id
  api_keys.yaml         API keys (SHA-256 hashes; created by scripts/api_keys.py)
core/                   Shared engine — imported by all three surfaces
screeners/              The models. Drop a file in, it auto-registers.
bot/                    Telegram bot + Gemini conversational layer
api/                    FastAPI REST service
scripts/                Operational CLIs (key management, usage, universe refresh)
tests/                  pytest suite (ledger, attribution)
training/               Offline model training (produces recovery_model_v4.pkl)
docs/                   This handbook, API docs, the served mockup
data/
  processed/            Large artefacts: .pkl, .parquet  (NOT in version control)
  *.db                  SQLite stores, created on first write
deploy/                 systemd unit + nginx site + install script (server only)

4. The data layer#

4.1 Main PKL — data/processed/idx_advanced_gross_model_multiindex.pkl#

The primary dataset. Daily OHLCV plus gross broker summary ("broksum") per ticker per day.

Group Columns
OHLCV Open, High, Low, Close, Volume
Regular market reg_value
Accumulation accum_top3_val, accum_top5_val, dist_top5_val
Flow bandar_net_flow, foreign_net_val, retail_net_val
Concentration top1_dominance, top3_avg_buy_price
Negotiated market nego_val, nego_avg_price, nego_premium
Derived signals markup_signal, bandar_strength, smart_money_divergence

"Bandar" is Indonesian market slang for a large operator moving a stock. bandar_net_flow is the net value accumulated by the top brokers — the platform's core edge signal.

Coverage characteristics worth knowing: on a healthy day roughly 921 of 962 tickers have a Close, ~830 have non-zero reg_value (i.e. actually traded), and ~105 have negotiated-market activity. Tickers with no trading still appear as rows with zeros — absence of a row is not how "didn't trade" is represented.

4.2 Broker detail — data/processed/broker_detail.parquet#

Per-broker, per-ticker, per-day transaction detail. This is the granular source the broksum columns above are aggregated from.

Read it with pyarrow filters, never fully — core/data_provider.py:load_broker_detail pushes ticker/date/market predicates down to the file.

4.3 Recovery model — data/processed/recovery_model_v4.pkl#

This is a trained model, not data. ~991 MB. Loaded by Model E (recovery_v4) to predict price trajectories. Unlike the other models, which retrain on every run, Model E loads this pre-trained bundle.

Regenerate with training/train_recovery.py. Note that its age matters: a bundle trained months ago scores today's prices with months-old parameters.

4.4 SQLite stores — data/*.db#

Five stores, each owned by one module. All created lazily on first write.

File Owner Tables
usage.db core/usage_store.py usage_events, consents
user_state.db bot/user_state_store.py watchlist, portfolio, alerts, user_prefs, signal_alert_log
chat_history.db bot/chat_store.py messages
feedback.db bot/feedback_store.py feedback
broadcast.db bot/broadcast_store.py broadcast_picks

House pattern for all of them: flat module functions, no ORM, connection opened and closed per call, CREATE TABLE IF NOT EXISTS on connect as the migration strategy, every function wrapped in try/except returning a benign default.

usage.db is the exception and deliberately so — it is the only store written by two processes (the API and the bot), so it uses WAL, a busy timeout, and PRAGMA user_version migrations. See §9.

4.5 Config files#

File Purpose Secret?
config/settings.py Paths, env var wiring, shared constants no
config/users.yaml Streamlit logins yes — plaintext passwords
config/telegram_users.yaml Telegram allowlist, int user id → {username, role} identities
config/api_keys.yaml API keys as SHA-256 hashes, mode 600 hashes only
.env Secrets + runtime flags, loaded by systemd EnvironmentFile yes

Nothing in the code loads .env. It only takes effect because the systemd units declare EnvironmentFile=. Running a process by hand without exporting those variables gives you defaults, not production behaviour — a real source of confusion when testing.


5. The feature engine#

core/feature_engine.py turns the 22 raw columns into roughly 60 features that the models consume. run_base_feature_pipeline(df) is the single entry point; everything else is a step inside it.

Family Function Produces
Volatility add_atr ATR_14
Momentum add_rsi rsi_14
Bands add_bollinger ma20, std_20, upper_band, lower_band, bb_width
Returns add_daily_returns daily_ret, rel_strength
Liquidity add_volume_features AvgValue_20d, AvgVolume_20d, vol_ratio, rvol, vol_dry_up
Broker flow add_bandar_features Intensity, Accum_Intensity, Markup_Smooth, Smart_Money_Ratio, Bandar_Flow_5d, Bandar_Accum_2W
Regime calculate_market_regime regime_score, mkt_volatility
Interactions add_interaction_features Flow_Divergence, Accum_Squeeze, Whale_Momentum
Risk add_overextension_risk dist_ma20_norm, Overextension_Risk
Crossovers add_golden_cross_features gc_ma5_ma20, gc_macd_line, gc_stoch_kd plus _days and _spread variants

Two things to know:

Computed features are cached in-process for one hour (screener_service._get_featured_data, _FEATURE_CACHE_TTL = 3600). The first call after a restart pays the full pipeline cost — measured at 30–45 seconds.


6. The models#

Models live in screeners/. Auto-discovery: screeners/__init__.py scans the directory on import, imports every .py, and registers any BaseScreener subclass by its metadata().id. Adding a model is dropping in a file — there is no registry to edit.

6.0 The contract#

class BaseScreener(ABC):
    @staticmethod
    def metadata() -> ScreenerMetadata   # id, name, description, category, version
    def render_sidebar(self) -> dict     # Streamlit widgets → params
    def run(self, params: dict) -> Any   # the actual analysis
    def render_results(self, results, params)   # Streamlit display

run() is the only method the bot and API call. The other two are Streamlit-only, which is why headless callers must install the Streamlit stubs first (§7.1).

Eleven models register; six are exposed over the API. The rule is mechanical: a model appears in the API if and only if it has an entry in screener_service.PARAM_SCHEMAS, because without a schema there is no validatable contract.

id Name Category API File
regime_adaptive_v6 Model A: Regime Adaptive swing ✅ regime_adaptive_v6.py
gann_classic_v7 Model B: Cycle Analysis swing ✅ gann_classic_v7.py
gross_alpha_v10 Model C: General Purpose swing ✅ gross_alpha_v10.py
confluence_filter Model D: Confluence Filter swing ✅ confluence_filter.py
recovery_v4 Model E: Portfolio Recovery recovery ✅ recovery_v4.py
model_f_combined Model F: Combined ML + Signals swing ✅ model_f_combined.py
gann_planner_v9 Model G: Dynamic Cycle-AI Planner position ✖ gann_planner_v9.py
backtest_lab Backtest Lab experimental ✖ backtest_runner.py
confluence_backtest Backtest D experimental ✖ confluence_backtest.py
model_f_backtest Backtest F experimental ✖ model_f_backtest.py
recovery_backtest Backtest E experimental ✖ recovery_backtest.py

Note backtest_lab lives in backtest_runner.py — the only model whose id does not match its filename.

6.1 The shared swing pipeline#

Models A, B, C, F and G are variations on one pipeline. Understand this once and each model is then just its differences.

  1. Load the cached feature frame (_get_featured_data, §5).
  2. Filter liquidity — min_price, min_val/min_daily_val, min_vol drop names too thin to trade. Typically ~830 of 962 tickers survive.
  3. Apply the FCA filter — apply_fca_filter() removes stocks under IDX's Notasi Khusus special-notation watchlist.
  4. Label history with core/backtester.create_order_aware_labels(). For each historical row: within hold_days, did price hit +target_gain before −stop_loss? The "order-aware" part matters — a naive implementation that checks max-gain and max-loss independently will label a trade a winner even when the stop was hit first. The docstring calls this out as a fixed temporal ordering bug.
  5. Train an XGBoost classifier on the labelled history, excluding the analysis date. This happens on every single run — there is no model cache. It is the reason a run costs 9–16 s and why the API is asynchronous.
  6. Score the analysis date → Probability per ticker (0–100).
  7. Filter and rank by the model's own strategy, cut to max_picks.
  8. Size positions — most models compute suggested allocation from capital, max_alloc, risk_cap and the measured win rate/payoff.

Each model also returns win_rate and payoff from its own test split, which is the honest way to read its picks: a 70% probability from a model with a 0.15 win rate means something very different from the same number at 0.55.


6.2 Model A — regime_adaptive_v6 (Regime Adaptive, v6.1)#

The idea: a pattern that works in a bull market does not work in a bear market, so train only on history that looks like today.

What makes it different. Where other models train on all available history, Model A computes the current market regime_score and then selects training rows whose own regime_score falls within ±0.20 of it:

tol = 0.20
regime_mask = (valid['regime_score'] >= current_regime - tol) & \
              (valid['regime_score'] <= current_regime + tol)

The classifier therefore learns "what worked in conditions like these" rather than a blended average across every regime. The trade-off is a smaller training set — in a rare regime there may be little matching history, and the model is correspondingly less confident.

It also adds two interaction features the others do not use: Accum_Squeeze (accumulation happening while volatility compresses) and Whale_Momentum (large broker flow aligned with price momentum).

Key parameters: capital, min_price, min_val, min_vol, target_gain (0.10), stop_loss (0.05), hold_days (5), max_alloc, min_intensity, min_probability.

Returns: df_result, diag, win_rate, payoff, train_positives, train_total. Results include the regime label so the UI can show the market state the picks were produced under.

When it suits: trending markets with a clear direction. The model's own guidance (core/model_docs.py) tells users that results are best when regime_score is above +0.20 or below −0.20, and to consider waiting when the market is sideways.


6.3 Model B — gann_classic_v7 (Cycle Analysis, v7.7)#

The idea: price moves in cycles off pivot lows, and both the distance from a pivot and the time elapsed since it carry information.

What makes it different. It layers Gann-style cycle features on top of the shared pipeline before training:

Feature Meaning
Days_Since_Low / Days_Since_High Bars since the last 60-day pivot
Cycle_PnL Gain from the pivot low — where in its run the stock is
Gann_Angle_Support Pivot low + Days_Since_Low × ATR/2 — a rising support line
Dist_Gann_Angle How far price sits above/below that line
Gann_Res_360 (√pivot_low + 1)² — the classic square-of-nine resistance
Low_Cycle_90 / High_Cycle_90 Position within a 90-day cycle, as a modulo

Pivots are detected as rolling 60-day extremes, then cumsum groups the bars belonging to each cycle so "days since" is computed per cycle rather than globally.

Key parameters: adds enable_gann (bool), gann_mode (Bullish/Bearish) and lookahead to the standard set. Defaults are wider than the other swing models — target_gain 0.15, stop_loss 0.08, hold_days 20 — because cycle setups are expected to take longer to play out.

When it suits: longer holds, and stocks with genuinely cyclical behaviour (commodities, cyclical blue chips). Its own documentation says to skip it in choppy markets, where cycles are not well defined.


6.4 Model C — gross_alpha_v10 (General Purpose, v10.0)#

The idea: the broker summary — who is accumulating, how concentrated it is, whether it diverges from price — is the edge. This is the platform's "bandarmology" model and its general-purpose default.

What makes it different. It adds a volume/volatility feature block on top of the base pipeline:

Feature Meaning
avg_price_{3,5,10,20}d, avg_vol_{…}d Multi-period trend filters
rvol Volume relative to its 20-day mean
vol_dry_up 20-day mean ÷ 60-day mean — volume drying up before a move
Volatility_20d Rolling return volatility
base_tightness 1 / volatility — a tight base scores high

Combined with the base broker features (bandar_net_flow, Accum_Intensity, Smart_Money_Ratio, Flow_Divergence), the model is looking for quiet accumulation in a tight base rather than momentum that has already happened.

_generate_context() turns each pick's feature values into a human-readable sentence, which is what the Telegram bot shows alongside a pick.

Key parameters: capital, max_alloc (0.25), risk_cap (0.05), target_gain (0.10), stop_loss (0.05), hold_days (5), min_intensity (0.05), min_price (200), min_daily_val (1e9), min_daily_vol, vol_price_days, filter_mode, retrain, min_probability.

A note on min_intensity: this one filter is frequently why Model C returns nothing. On a small or quiet universe it can eliminate every candidate before probability is even considered — the funnel will show after_intensity: 0. Loosening it is usually the right first move, and the empty-results explainer says so.

When it suits: the default choice for 5-day swings. It is one of the two models the daily broadcast runs.


6.5 Model D — confluence_filter (Confluence Filter, v1.0)#

The idea: no machine learning at all. Look for stocks where several classic technical signals fire together or in close sequence.

The sequence it hunts for, in the order these normally occur:

  1. Stochastic %K crosses up out of oversold — earliest, catches the reversal
  2. MACD histogram flips positive — momentum shifting
  3. MACD line crosses above signal — momentum confirmed
  4. MA5 crosses above MA20 — trend confirmed

_detect_signals() computes each as a boolean per bar; _count_recent_signals() counts how many fired within the lookback window; stocks with at least min_signals pass, with volume above average as confirmation.

Because there is no training step it is the fastest model — a few seconds rather than fifteen — and it is fully deterministic: the same inputs always give the same picks.

Key parameters: lookback (5), min_signals (2), stoch_period, stoch_oversold, macd_fast/slow/signal, ma_fast/ma_slow, min_vol_ratio, and use_stoch / use_hist / use_macd / use_ma to enable each signal independently.

Returns: df_result, diag, total_scanned, total_hits, excluded_seller, signal_distribution — the last being a histogram of how many stocks had 0, 1, 2, 3 signals, which is useful for calibrating min_signals.

When it suits: when you want an explainable, auditable screen with no model risk. It is the second model the daily broadcast runs.


6.6 Model E — recovery_v4 (Portfolio Recovery, v4.1)#

Completely different from every other model. It does not screen the market — it analyses positions you already hold, most of which are losing, and answers "what is the realistic recovery path?"

It does not train. It loads the pre-trained bundle data/processed/recovery_model_v4.pkl (~991 MB) and runs inference. This is why it is both the fastest model in warm steady state and the only one that will not run at all without its artefact.

What it predicts, per position, at a horizon of 10/20/30/40 trading days:

make_decision() turns that into advice. Its central calculation is the breakeven requirement: a position down 41.8% needs 0.418/(1−0.418) = +71.8% to get back to cost. The function compares that requirement against the model's own p75 upside and produces a verdict — the reason strings users see, such as "Recovery unlikely: need +72%, P75 only +8%". It also decides an optimal_exit_days window and a plan (hold / average down / cut).

Input shape — each position takes either form, not both:

{"ticker": "GOTO", "pnl_input": -41.81}          // percent
{"ticker": "GOTO", "avg_price_input": 96.24}     // your average cost

Returns: positions, horizon, analysis_date. Each position carries outlook (BULLISH/NEUTRAL/BEARISH), reason, plan (target, stop_loss, exit_window, and a large explanation dict) and prediction (trajectory, probabilities, quantiles).

Two sharp edges:

Naming inconsistency to be aware of: recovery_v4.py:27 defines TRAJ_CLASSES with CLIMB, but config/settings.py:123 still lists the older BOUNCE_FADE. Line 223 handles both for backward compatibility with older PKLs, so nothing breaks — but settings.py is the stale one, and CLIMB is what the API actually returns.


6.7 Model F — model_f_combined (Combined ML + Signals, v1.0)#

The idea: Model C knows probability, Model D knows confluence. Combine them.

Model F computes both — v10 features and Model D's signal detection (it imports _detect_signals and _count_recent_signals directly from confluence_filter) — then applies one of four selection strategies:

strategy Logic
Gated Filter to confluence_count ≥ min_signals, then rank by probability. Quality over quantity
Hybrid Score Blended score: prob_weight × prob + (1−prob_weight) × signals, keeping anything that passes probability or has ≥1 signal
Intersection Strict: must pass both min_prob and min_signals. Fewest picks, highest conviction
Inverse Signal The default and the backtest winner. score = prob_weight × (prob/100) − signal_penalty × (signals/4)
Compare All Runs all four and returns them side by side, plus an overlap count

Inverse Signal deserves explanation because it looks wrong at first glance: it penalises confluence. The reasoning is that by the time four signals have fired, the move is visible to everyone and you are late. It prefers high model probability with few visible signals — accumulation the market has not yet priced. Backtesting made it the default.

Compare All is genuinely useful during analysis: a stock appearing in all four strategies is a consensus pick.

Key parameters: strategy, prob_weight (0.6, range 0.3–0.8), signal_penalty (0.3, range 0.0–0.5, only affects Inverse Signal), min_signals, min_probability, lookback (2 — fresh signals only), target_gain, stop_loss, hold_days, max_picks, min_price (500), min_daily_val (1e10 — notably stricter than Model C).

Returns: picks (not df_result), target_date, win_rate, payoff, n_candidates, diag. Under Compare All it also returns gated, hybrid, intersection, inverse_signal and overlap.

Caveat in the code's own words: it "does NOT broadcast to the daily channel — runs only in Streamlit for exploration." It is nonetheless the model the FAIR mobile client uses, via the API.


6.8 Model G — gann_planner_v9 (Dynamic Cycle-AI Planner, v9.4)#

Not exposed over the API — it has no PARAM_SCHEMAS entry. Reachable from Streamlit only. Adding a schema would publish it.

The most elaborate model in the repo. On top of Gann cycle features it adds:

Category position rather than swing: it is built for longer holds.


6.9 The backtest models#

Four experimental entries, all excluded from the API. They exist to validate the live models rather than to produce picks.

id What it validates
backtest_lab Walk-forward harness for the ML models generally (backtest_runner.py)
confluence_backtest Model D: recomputes signals at each historical eval date, simulates the resulting trades. No training involved
model_f_backtest Model F: a custom walk-forward loop that computes v10 features and D signals once, then trains at each eval date and compares all strategies
recovery_backtest Model E — and this one is not a trade backtest. Model E predicts trajectory classes and outlook, not buy/sell signals, so this audits whether those predictions correlate with realised outcomes

The shared engine is core/backtester.py (661 lines), which also provides create_order_aware_labels() used by every live ML model.


6.10 Result shapes are not uniform#

This catches people, so it is worth stating plainly:

Model Picks key Ticker key Date key
A, B, C, D df_result Ticker target_date / data_date
F picks Ticker target_date
E positions ticker (lowercase) analysis_date

core/usage_store.summarize_result() and api/views.py both probe all three defensively — copy that approach rather than assuming one.

Raw results also mix display strings with numbers. df_result rows carry "Last Close": "8,050" and "Daily Val": "139.4B" next to genuine floats, because they are rendered straight into Streamlit tables. The API's view=compact exists to give clients real numbers instead.

7. Surface 1 — the Streamlit app#

app.py, served on 127.0.0.1:8508, published at multiscreener.qomu.ai.

Flow: core/auth.py gates on config/users.yaml → a sidebar dropdown built from the screener registry → the selected model's render_sidebar() collects params → run() → render_results().

Notes: - Login is a plaintext password compare. See Known issues. - core/model_docs.py supplies the plain-language "ℹ️ Tentang model ini" text. - core/empty_results_explainer.py renders the "nothing passed your filters" panel with suggestions for which filter to loosen. The API reuses its suggestion logic for the same purpose. - Streamlit builds params through each screener's own render_sidebar() and never touches PARAM_SCHEMAS — so API schema changes cannot affect the UI.

7.1 Running screeners headlessly#

screeners/*.py call st.toast, st.spinner, st.session_state and friends directly. To run them outside Streamlit, core/screener_service._install_streamlit_stubs() monkey-patches those into no-ops and turns @st.cache_data into a real in-memory cache.

Every headless entry point must call it before importing screeners. The bot, the API and every script in scripts/ do.


8. Surface 2 — the Telegram bot#

bot/telegram_bot.py, long-polling, no webhook.

Identity#

The canonical id is the integer Telegram user id. config/telegram_users.yaml maps it to {username, role} with roles user < pro < admin. The file is hot-reloaded on mtime, so adding a user takes effect without a restart. If the file is empty the bot runs in open mode and allows everyone — worth knowing before you clear it.

Users added by @handle get a pending_<username> placeholder key, upgraded to their real integer id on first message by _backfill_user_id.

Conversational layer#

bot/gemini_parser.py is the brain. A user message goes to Gemini with a set of declared tools; Gemini decides which to call; _execute_tool_call dispatches to the service layer. Tools cover running screeners, quotes, broker flow, nego activity, turning points, watchlist and portfolio management, and feedback.

Without a GEMINI_API_KEY it degrades to _fallback_chat, a keyword/regex parser that can still run screeners.

Scheduled jobs#

Job Time (WIB) What
daily_broadcast 08:00 Runs Models C and D, posts top picks to the channel
user_daily 07:30 Per-user DMs: turning-point alerts, price alerts, digest
usage_prune 03:30 Trims the usage ledger past its retention window

Both pushes are gated by PUSH_ENABLED. Set PUSH_ENABLED=0 in .env and neither is scheduled at all — the scheduler registers zero push jobs. Admin commands /broadcast, /broadcasttest and /digest are deliberate manual actions and are not gated.

Commands#

30 registered. User-facing: /start /help /list /clear /watch /unwatch /watchlist /port /unport /portfolio /alert /alerts /delalert /signals /whoami. Admin-only, DM-only: /adduser /adduserid /removeuser /users /confirm /exportfeedback /feedbackstats /usage /chathistory /broadcast /broadcasttest /digest /pick /winrate /winrateat /dm.


9. Surface 3 — the REST API#

api/server.py, uvicorn on 127.0.0.1:8600, published at dev.multiscreener.qomu.ai. Full reference in API.md.

Layout#

Everything machine-facing is under /api; human-facing pages sit at the root.

/api/v1/models        /api/v1/jobs      /api/v1/data
/api/v1/usage         /api/v1/consent   /api/v1/admin/usage
/api/health           /api/docs         /api/openapi.json
/guide  /guide/integration  /guide/reference  /mockup

Module map#

File Role
server.py App, middleware, host allowlist, legacy redirects, system routes
auth.py API keys, scopes, rate limiting, X-End-User validation
jobs.py Background run execution, two-worker pool, TTL eviction
routes_models.py Catalogue, schemas, run submission, the analysis-date pin
routes_jobs.py Poll jobs; two-tier live/ledger contract
routes_data.py 14 market-data endpoints
routes_usage.py Tenant usage + separate admin router
routes_consent.py Disclaimer gate
views.py view=compact card shapes
validation.py Rejects unknown/out-of-range params
serialization.py numpy/NaN/Timestamp → strict JSON
errors.py One error envelope + request ids
docs_pages.py Renders docs/*.md at /guide/*

Two identities#

Authorization: Bearer idxs_live_…   ← the consuming APPLICATION
X-End-User: nasabah_00471           ← the end CUSTOMER (required on runs)

Keys are random 256-bit secrets stored as SHA-256 digests — not bcrypt, because high-entropy secrets cannot be dictionary-attacked and the API hashes on every request. Scopes: models:read, models:run, data:read, usage:read, usage:admin. The key file is hot-reloaded on mtime, so issuing and revoking take effect within a second with no restart.

Why runs are asynchronous#

A run retrains the model: 9–16 s warm, 30–45 s cold. POST .../runs returns 202 with a job id to poll, or 200 inline if ?wait=N (max 60) was satisfied. Concurrency is capped at 2 workers because each run pins a large DataFrame and trains a model — more workers make every run slower, not faster.

The usage ledger#

core/usage_store.py records every model run from all three channels into one table, so "who ran what, with which parameters" is a single query. Written by two processes, hence WAL, a 5 s busy timeout, and PRAGMA user_version migrations.

Writes are best-effort and fail silently by design — a ledger problem must never break a model run. That makes GET /api/health → usage.writable the only way to notice it has stopped working. Reads deliberately do raise: losing a write is acceptable, lying about history is not.


10. Running it locally#

python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt

# Data artefacts are NOT in version control. Either copy the real ones into
# data/processed/, or use the sample bundle (see export/README.md).

# Streamlit
.venv/bin/streamlit run app.py

# REST API
.venv/bin/python scripts/api_keys.py create local-dev \
    --scopes models:read,models:run,data:read
.venv/bin/uvicorn api.server:app --host 127.0.0.1 --port 8600

# Telegram bot (needs TELEGRAM_BOT_TOKEN, and GEMINI_API_KEY for conversation)
export TELEGRAM_BOT_TOKEN=... GEMINI_API_KEY=...
.venv/bin/python -m bot.telegram_bot

# Tests
.venv/bin/python -m pytest tests/ -q

tests/test_usage_store.py runs in under a second because core/usage_store.py is import-light by design. tests/test_attribution.py takes ~30 s because importing the bot pulls in the Gemini and Telegram clients.

Environment variables#

Variable Default Effect
TELEGRAM_BOT_TOKEN — Bot auth. Without it the bot will not start
GEMINI_API_KEY — Conversational layer; absent → regex fallback
PUSH_ENABLED 1 0 disables both scheduled pushes
BROADCAST_CHANNEL_ID a channel id Comma-separated targets; empty also disables the broadcast
BOT_MIN_PROBABILITY 65 Default probability floor for bot picks
IDX_MAIN_PKL / IDX_RECOVERY_PKL / IDX_BROKER_DETAIL filenames Override which artefact is loaded
IDX_USAGE_DB data/usage.db Ledger location
IDX_USAGE_RETENTION_DAYS 180 0 = keep forever
IDX_API_KEYS_PATH config/api_keys.yaml Key file
IDX_API_JOB_WORKERS 2 Concurrent model runs
IDX_API_ALLOWED_HOSTS (empty) Comma-separated Host allowlist; empty = no restriction
IDX_API_PIN_ANALYSIS_DATE (empty) Clamp every run to this date — integration environments
IDX_API_CORS_ORIGINS (empty) CORS origins; empty = off

11. Deployment#

The server#

One Ubuntu box, 103.150.100.165, hostname idx-experiment. Everything lives in /home/prasma/multiscreener with a single .venv. Three systemd units, all running as user prasma:

Unit Binds Published as
multiscreener-streamlit 127.0.0.1:8508 multiscreener.qomu.ai
multiscreener-telegram — (outbound polling) —
multiscreener-api 127.0.0.1:8600 dev.multiscreener.qomu.ai

Each unit uses EnvironmentFile=/home/prasma/multiscreener/.env and Restart=always. Nothing binds a public interface — nginx terminates TLS and proxies to localhost. An API key over plain HTTP is a key in the clear.

Deploying a change#

scp api/server.py prasma@multiscreener.qomu.ai:/home/prasma/multiscreener/api/
sudo systemctl restart multiscreener-api

Python loads modules once at import. Changing a file on disk does nothing to a running process — a restart is always required, and until you do it the on-disk and in-memory code differ. With Restart=always an unattended crash will pick up the new code at an arbitrary hour, so restart deliberately rather than leaving the gap open.

Never overwrite: config/*.yaml, .env, data/. Production accumulates users and state there.

Standing up the API from scratch#

deploy/install-api.sh (run with sudo) installs the systemd unit, verifies /api/health locally, installs the nginx site, runs nginx -t before every reload so a mistake cannot take down the other vhosts, then issues the certificate with certbot --nginx --redirect. Idempotent.

Sudo scope#

Passwordless sudo on this box is narrow:

(ALL) NOPASSWD: /bin/systemctl restart multiscreener-*,
                /bin/systemctl status  multiscreener-*,
                /bin/journalctl -u     multiscreener-*

Service restarts need no password; nginx, certbot, writing systemd units and reading /etc all do. This is why services are named multiscreener-* — the prefix is what makes them restartable unattended.


12. Operations#

Health#

curl -s https://dev.multiscreener.qomu.ai/api/health | jq
#  data_loaded      false right after a restart → first request pays the cold start
#  usage.writable   false means ledger writes are failing SILENTLY
#  jobs.by_status   queued piling up means runs are backing up

sudo systemctl status multiscreener-{api,telegram,streamlit}
sudo journalctl -u multiscreener-api -n 100 --no-pager

Common tasks#

# API keys
python scripts/api_keys.py create acme --scopes models:read,models:run
python scripts/api_keys.py list
python scripts/api_keys.py disable acme          # effective within 1s, no restart

# Usage
python scripts/usage.py stats
python scripts/usage.py top --by model --days 30
python scripts/usage.py recent --channel telegram
python scripts/usage.py prune --days 180 --yes

# Refresh the ticker universe (manual: IDX sits behind Cloudflare)
python scripts/refresh_idx_universe.py --manual-tickers GetSecuritiesStock.json

Turning pushes off#

sed -i 's/^PUSH_ENABLED=.*/PUSH_ENABLED=0/' .env
sudo systemctl restart multiscreener-telegram
# verify: the scheduler should log ZERO "Added job" lines for pushes
journalctl -u multiscreener-telegram --since "-2 min" | grep "Added job"

Data freshness#

Every result carries the trading date it reflects (as_of, target_date, analysis_date). Check staleness with:

import pandas as pd
from config.settings import Settings
df = pd.read_pickle(Settings.MAIN_PKL_PATH)
print(pd.to_datetime(df.index.get_level_values('Date')).max())

A healthy day has ~921 priced tickers and ~830 with non-zero reg_value. If reg_value > 0 drops far below that, the scrape was partial even though the file exists and the date looks current.


13. Known issues and sharp edges#

Ordered by how likely they are to hurt.

  1. Plaintext passwords in config/users.yaml. Streamlit login compares with ==. The module docstring has said "hash with bcrypt" since the beginning. Anyone who reads that file has every account.

  2. The bot token is logged in plaintext. httpx logs request URLs at INFO and the token sits in the path, so every getUpdates line leaks it to anyone who can read the journal. One-line fix: logging.getLogger("httpx").setLevel(logging.WARNING).

  3. Memory headroom is thin. 8 GB total, no swap. The API alone held ~1.6 GB and the bot ~0.9 GB before either loaded the 991 MB recovery bundle. Two concurrent Model E runs could invoke the OOM killer, which will not necessarily pick the process that caused it. Mitigations: add swap, set MemoryMax on the API unit, or drop IDX_API_JOB_WORKERS to 1.

  4. The ingestion pipeline is external and unreliable. It lives in a Colab notebook outside this repo. Observed failures: 2026-09-07 negotiated-market data almost entirely missing; 2026-09-16 broker summary covering 206 of ~835 tickers; 2026-09-17 onward no run at all. Nothing in this repo monitors it. A partial day looks completely normal from the file system.

  5. dev.multiscreener.qomu.ai is not a dev environment. It shares the directory, the virtualenv, the data files and the box with production. The name promises isolation that does not exist.

  6. Model E's bundle ages silently. It is loaded, not retrained, so its predictions quietly reflect whenever it was last trained while the prices it scores are current. Nothing surfaces that age.

  7. No schema versioning except in usage_store. The other four SQLite stores use CREATE TABLE IF NOT EXISTS, which silently no-ops against an older table. Adding a column there needs a manual ALTER.

  8. nginx routing depends on filename order. No block declares default_server for 443, so an unmatched hostname lands on whichever site loaded first. The API's IDX_API_ALLOWED_HOSTS guards against this from the application side; the nginx-level fix is still outstanding.


14. Glossary#

Term Meaning
IDX Indonesia Stock Exchange
broksum Broker summary — aggregated per-broker buy/sell per ticker per day
bandar Market slang for a large operator moving a stock; bandar_net_flow tracks it
nego Negotiated market — off-order-book block trades, distinct from REG
FCA Notasi Khusus — IDX's special-notation watchlist of flagged stocks
confluence How many independent technical signals agree on a stock
regime Market-wide state: bullish / sideways / bearish, as regime_score
nasabah Indonesian for "customer" — used as the end-user id in API examples
WIB Western Indonesian Time, UTC+7 — all schedules are in WIB
Source: docs/HANDBOOK.md in the idx-screener repository. This page is rendered from that file, so the repo stays the single source of truth.