Commit graph

7 commits

Author SHA1 Message Date
Peter Goodhall
f01599e6a5 Add indexes and batch processing for uploads
Improve query performance and memory usage by adding a migration to create several indexes and converting various upload/reporting flows to batched processing. Changes include:

- Add migration 245 to create indexes: idx_mode, idx_sat_name, idx_continent, idx_qsl_stats (composite), idx_hrdlog_upload, idx_qrz_upload to speed common queries.
- Bump migration version to 245 in config.
- Hrdlog and Qrz controllers: switch to fetching QSOs in LIMIT/OFFSET batches, process updates in chunks, add logging for batch progress, and use break 2 to stop outer loops on invalid API keys to avoid continuing work.
- Logbook_model:
  - get_hrdlog_qsos and get_qrz_qsos now accept limit/offset and use ORDER BY/LIMIT/OFFSET for batching.
  - Replaced a subquery with a JOIN for grid-based filtering to improve performance.
  - get_modes rewritten to use query builder (distinct/select/order_by).
  - Bulk DXCC updates use update_batch instead of per-row queries.
  - calls_without_station_id limited to 500 results to avoid UI hangs.

Overall goal: prevent memory exhaustion with large datasets and improve database query performance.
2026-02-11 10:59:24 +00:00
HB9HIL
2591d3aef6 fix error message bug 2023-12-16 13:44:39 +01:00
int2001
28fd5d5978
Set status if no QSO found in upload-loop 2023-12-16 12:31:33 +00:00
HB9HIL
e71f782a83 fixed success and error messages 2023-12-16 11:42:56 +01:00
HB9HIL
fc793a1160 added hrdlog_username 2023-12-15 23:04:34 +01:00
int2001
58e0a57f5e Removed whole index-function at controller 2023-07-07 12:19:21 +00:00
int2001
6253650190 Controller, Migrationscript and View for HRDLog 2023-07-07 11:35:01 +00:00