Automating a Daily Watchlist: A Scheduled Script That Fetches Data, Computes Signals and Writes a Report, plus API-Key and Rate-Limit Pitfalls

What you will learn

• How to restructure notebook cells into a script with functions, a main() entry point and a configuration file.
• How to keep API keys out of the code and handle rate limits with waiting and exponential back-off.
• How to write a report that is useful at 07:30 in the morning: a ranked table, a few flags and a timestamp.
• How to schedule the script with cron or Windows Task Scheduler, and what to log so that silent failures do not go unnoticed.

Explanation

A notebook is for exploring; a script repeats the same job every day without you watching. The indicators (lesson 6) and signal logic (lessons 7 and 8) stay the same; everything around them changes.

Structure of the script

Split the work into functions with one job each: load_config(), fetch(ticker), compute_signals(df), build_report(rows) and main(). The guard if __name__ == '__main__': main() lets you run the file and also import its functions into tests. Keep the watchlist, parameters and output folder in config.yaml or config.json so they change without editing code.

API keys and secrets

Never write a key into source code or a notebook you might share or commit to Git. Read it from an environment variable or a .env file with os.environ['DATA_API_KEY'] (the python-dotenv package loads .env), and list .env in .gitignore. Treat the key like a password and rotate it if it leaks.

Rate limits, retries and back-off

A rate limit is the maximum number of requests a provider allows per minute or day; exceeding it returns an HTTP 429 error or, worse, empty data with no error. Pace the loop with time.sleep(60 / max_per_minute). For temporary failures (429, 500, time-outs) retry with exponential back-off: wait 1 s, 2 s, 4 s, 8 s, then give up and log the ticker as missing. Never retry in a tight loop; that is how keys get banned.

• def fetch_with_retry(ticker, tries=4, base=1.0):
• for n in range(tries):
• try:
• return fetch(ticker)
• except TransientError as e:
• wait = base * 2 ** n
• log.warning('%s: %s, retry in %.0fs', ticker, e, wait)
• time.sleep(wait)
• log.error('%s: giving up', ticker)
• return None

Cache raw downloads (data/raw/TICKER_YYYY-MM-DD.parquet): a second run on the same day reads the file instead of spending quota, and you keep an audit trail of what the script saw.

Validation before signals

Data arrives broken more often than you expect: a missing last bar, a stale date, a close of 0, an unadjusted split. Check that the last date equals the expected trading date, that the last 20 rows have no NaN, and that today's absolute return is below a sanity threshold such as 50 %. Failures go to a "data problems" section of the report instead of producing a buy signal on a bad print.

The report

Write a Markdown or HTML file with the run timestamp and data date, a table sorted by your ranking (for example distance from the 50-day SMA in ATR units), the new signals since yesterday (compared with the previous day's saved signal file) and the data problems. Save it as reports/watchlist_2026-10-30.md so history is kept; charts per flagged ticker (lesson 10) are optional.

Scheduling and logging

On Linux and macOS, cron runs commands on a timetable: 30 6 * * 1-5 /path/venv/bin/python /path/watchlist.py >> /path/logs/cron.log 2>&1 runs at 06:30 on weekdays; on Windows, Task Scheduler does the same through a dialog. Use the full path to the virtual environment's Python (lesson 1), because schedulers do not activate environments. Log with the logging module to a file: start, tickers fetched, signals found, end. No entry for today means the scheduler failed. Computers sleep, networks drop, endpoints change; a script you do not check is not automation.

Worked example

Hypothetical example: a watchlist of 40 European and US shares, a free data plan with a limit of 5 requests per minute and 500 per day (invented figures; check your provider's current terms).

1. Pacing. 40 tickers at 5 per minute = 40 / 5 = 8 minutes minimum. The script sleeps 12 seconds between calls. Scheduling at 06:30 means data is ready by about 06:40.
2. Quota. One run uses 40 of 500 daily calls (880 a month over 22 trading days). A bug that reruns every 10 minutes would burn 240 calls an hour and hit the cap in about two hours; a disk cache removes that risk.
3. Back-off. Two tickers return HTTP 429 on first try. Retries after 1, 2 and 4 seconds succeed on the second attempt; worst-case added time is 1 + 2 + 4 + 8 = 15 seconds per ticker, so the run still finishes within the 10-minute budget.
4. Report. Rule: "close crosses above the 50-day SMA with RSI(14) below 60". Yesterday's file had 3 active signals; today's run finds 5, of which 2 are new. The report lists the 2 new names first, the 3 continuing ones, and 1 ticker under "data problems" because its last bar is two days old.

Excluded: this says nothing about whether the signal is profitable; it only makes the daily preparation reliable and repeatable.

Common mistake

Practice activity

On paper: your provider allows 30 requests per minute and 2,000 per day. You want to run a 150-ticker watchlist once at 06:00 and refresh 20 priority tickers every 30 minutes between 09:00 and 17:30 (CET). (a) What minimum sleep between calls respects the per-minute limit? (b) How many calls does a full day use? Is it inside the daily quota? (c) The 06:00 run reports 150 tickers fetched but all signals identical to yesterday. Name two checks that would tell you whether the data is fresh.

Key takeaways

• Scripts need functions, a config file, a main() guard and logging; notebooks do not.
• Keep keys in environment variables or .env; pace requests to the limit and retry with exponential back-off, then give up and log.
• Validate dates and row counts before computing signals; cache raw downloads.
• Schedule with cron or Task Scheduler using the full path to the virtual environment's Python, and check the log daily.

Next step

Lesson 13 assembles everything into a complete research-notebook project, from data to report, and reviews it with a checklist against look-ahead bias and overfitting.