How to Avoid OutOfMemoryException When Concatenating Strings

"When I tried to write an answer saying ""It's better to use StringBuilder for string string concatenation,"" I made a code example, but there were so many cases that I got Out of MemoryException even if I used StringBuilder."

If there are too many cases to handle everything in memory, I think it's better to change your mind to process every line.

In this way, even if there are more than a million cases, there will be no errors as long as disk space permits.
分散 It may be best to store it in DB with distributed processing, but it is not covered here because it requires setting up the environment.

Below is a sample code that temporarily writes a file called C:\tmp123456789.txt, processes it line by line, and then deletes the file.
In my environment, the output took about 30 seconds for every 10,000 lines, but the entire process took about 80 seconds.

Imports System.IO

Module Module 1

    Sub Main()
        DimtmpFile="C:\tmp123456789.txt" 

        Dim stopwatch As New Stopwatch
        stopwatch.Start()

        Dimrnd As Random = New Random
        'Because of the large number of cases, StringBuilder is running out of memory.
        'Dimsb As StringBuilder = New StringBuilder
        'Open the file and output it sequentially
        Dim writer As New Stream Writer (tmpFile, False)
        'A mode in which data is automatically written when it is accumulated to a certain extent
        writer.AutoFlush=True

        '10 to the 5th power = 10,000
        Fori=1 To Math.Power(10,5)
            Dimlist As List (Of Single) = New List (Of Single)
            'line data creation
            For j=1 To 1000
                list.Add (CType(rnd.NextDouble(), Single))
            Next
            'Line data output (If you don't need to handle all the data at once, you can process each line here without using StreamWriter)
            writer.WriteLine (String.Join(", ", list))
        Next
        writer.Close()
        Dimmsg1 = String.Format ("temporary file creation time: {0} seconds", stopwatch.ElappedMillisconds/1000)
        Console.WriteLine(msg1)
        US>'If you want to output the file, that's it.

        'Read each line of the temporary file while processing
        Dim total As Double = 0
        Dim reader As New StreamReader (tmpFile)

        ' repeat until there are no more characters to read
        While (Not reader.EndOfStream)
            ' load files line by line
            Dimline=reader.ReadLine
            'Do something (add as a sample here)
            For Each Inline.Split(", ")
                total+=Single.Parse(s)
            Next
        End While
        reader.Close()
        'Cleaning up afterwards
        File.Delete(tmpFile)

        Dimmsg2 = String.Format ("Processing completion time: {0} seconds", stopwatch.ElapsedMilliseconds/1000)
        Console.WriteLine(msg2)
        Console.WriteLine("Total value is {0}",total)
        Console.ReadLine()
    End Sub

End Module

I will revise my answer based on yhata's suggestion.

If OutOfMemoryException occurs due to unlimited string combinations, it may be out of memory, as the exception indicates.The 32-bit application can use 2GB of memory, but not all of it can be used for data, so OutOfMemoryException occurs even if it is used less.
Using StringBuilder instead of String increases processing speed, but the amount of data that can be processed is rarely improved.In the end, it is necessary to drastically review the processing methods.At a minimum, you cannot repeat String joins indefinitely.
Inevitably, the String bond should be kept in a small amount, but if the string created is kept to approximately 40,000 characters or less, there is no memory difference between the String bond and the StringBuilder.
For example, if this is the case, it might be better to review it so that everything is handled one by one instead of all at once.In that case, one data is

Single numbers separated by commas and 1000 comma-separated strings

However, at this rate, even in the worst case, it can fit around 10,000 characters.

StringBuilder has excellent string editing performance, as yhata explained, because it is managed in a split manner so that even if a large string is created, it can be less than 85KB.However, this is an improvement in .NET 4 and cannot be enjoyed before .NET 3.5, so be careful.What you should do in that case is to estimate the string length in advance and set the StringBuilder.Capacity property.This allows pre-memory protection and eliminates sequential memory expansion when editing strings.