Exporting billion-row tables from ABAP

At some point you need ACDOCA — or MSEG, or CDPOS — out of the SAP system and into somewhere else. The obvious approaches fail in escalating order of expense: SE16 dies long before interesting row counts; SELECT ... INTO TABLE needs the full result in one work process's memory; and the hand-written background job runs for days, dies at row 800 million on a work-process restart, and leaves nothing restartable. A long-running job without resume is worse than useless — the recovery plan is "run it again and hope", and hope restarts the clock from zero.

ZTABLE_EXPORT_CSV closes exactly those three gaps. Single ABAP report, MIT, in the Datenrösterei repo. Three design decisions; everything else is detail.

One: a cursor, not an internal table

Solved in ABAP long before HANA existed: OPEN CURSOR + FETCH ... PACKAGE SIZE. The database holds the result set; ABAP holds one package.

OPEN CURSOR WITH HOLD lv_cursor FOR
  SELECT (gt_fieldlist) FROM (p_table)
  WHERE (gv_where_clause)
  ORDER BY (gv_orderby).

DO.
  FETCH NEXT CURSOR lv_cursor INTO TABLE <lt_data> PACKAGE SIZE p_pkg.
  IF sy-subrc <> 0. EXIT. ENDIF.
  PERFORM process_package USING <lt_data>.
  CLEAR <lt_data>.
ENDDO.

Table, field list, WHERE and ORDER BY are all dynamic (DDIC via DDIF_FIELDINFO_GET, package table via cl_alv_table_create). Memory is P_PKG rows, constant, at fifty thousand rows or 1.5 billion. The ORDER BY on the primary key isn't decoration — it's what makes decision three possible, and on HANA it's essentially free.

Two: chunked output

One 500 GB CSV is a liability: CG3Y won't move it, most tools choke on it, and corruption at byte 400 billion costs the whole file. The report rolls to a new file every P_CHUNK rows or every P_MAXGB gigabytes — the mode I actually use. Each chunk optionally carries its own header, so every file loads independently. Two-gigabyte chunks turned out to be the sweet spot: few enough files to manage, small enough that each one moves without drama.

Three: resume via keyset pagination

The decision that separates a tool from a demo. After each package the report overwrites a small checkpoint file: counters, plus the primary-key values of the last row written. Dies the job, you re-run with P_RESUM = 'X' and it builds a composite-key seek predicate:

" For keys (K1,K2,K3) with last values (V1,V2,V3):
"   ( K1 > 'V1' )
"   OR ( K1 = 'V1' AND K2 > 'V2' )
"   OR ( K1 = 'V1' AND K2 = 'V2' AND K3 > 'V3' )

Keyset, not OFFSET — OFFSET 800000000 makes the database produce and discard 800 million rows before the first one you want, a resume costing nearly as much as the run it replaces. The keyset predicate is an index seek to a position. Zero rows skipped, immediate.

Two honest caveats. It needs a real primary key (any transparent DDIC table has one). And the guarantee past the checkpoint is at-least-once: a crash mid-package rewrites that package's tail into the next file. If duplicates matter downstream, dedupe on the PK — you exported it anyway.

Details that bite three systems later

• Locale-independent numerics. ABAP's WRITE TO formats per the user's SU01 settings — on German systems, 1.234,56. That comma reaching a parser expecting points gives either a parse error (the good outcome) or a silently wrong number. The report never uses WRITE TO.
• Escaping only when needed. Semicolon-delimited, "-enclosed, enclosures doubled — but the escape path only runs for fields that actually contain a delimiter or line break. On a 518-column table you don't pay escaping on 517 clean fields.
• 2010-era ABAP by design. The systems that most need a bulk exporter are old ECC boxes being migrated or decommissioned; a tool leaning on modern syntax is useless exactly there. One caveat a reviewer caught before I did: the row/byte counters are int8, a 7.40 builtin — so the header's "7.02+" claim is optimistic. On a real EHP2 box, eight declarations become TYPE p LENGTH 8 and the rest activates as-is.

What it costs

Measured on S/4HANA 2023 against ACDOCA (44,818-row test set): ~2,700 rows/s at all 518 columns, ~26,000 rows/s at a 41-column selection. Extrapolated — labelled as such — a full-width 1.5-billion-row export lands around six and a half days; a 60-column selection around 23 hours. Field selection is by far the biggest lever.

The first version did 278 rows/s. The 9.7× to 2,700 was entirely ABAP-side: LOOP ASSIGNING, ASSIGN COMPONENT by index, a pre-allocated string table, CONCATENATE LINES OF, and a fast path for IS INITIAL fields — 70–90% of any ACDOCA row.

And the measurement that reframed the whole effort: package size has no effect on throughput. 20,000 rows per fetch or 200,000 — same speed. The database was never the bottleneck; HANA hands over packages faster than a work process can serialize them. My intuition said bulk extraction is database work. The profiler said it's a string-processing job with a database attached. The profiler wins.