C# read huge text files

Shahin Khorshidnia Jun am. Add your solution here. OK Paste as. Treat my content as plain text, not as HTML.

Existing Members Sign in to your account. This email is in use. Do you need your password? Submit your solution! When answering a question please: Read the question carefully.

Understand that English isn't everyone's first language so be lenient of bad spelling and grammar. If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome. Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question. Let's work to help developers, not make them feel stupid.

Related Questions. New data is appended to the end of the buffer if room is available; otherwise, a new, larger buffer is allocated, data from the original buffer is copied to the new buffer, then the new data is appended to the new buffer. The performance of a concatenation operation for a String or StringBuilder object depends on how often a memory allocation occurs.

A String concatenation operation always allocates memory, whereas a StringBuilder concatenation operation only allocates memory if the StringBuilder object buffer is too small to accommodate the new data. Consequently, the String class is preferable for a concatenation operation if a fixed number of String objects are concatenated. In that case, the individual concatenation operations might even be combined into a single operation by the compiler. A StringBuilder object is preferable for a concatenation operation if an arbitrary number of strings are concatenated; for example, if a loop concatenates a random number of strings of user input.

That means huge allocation of memory, what becomes large use of swap files system, that simulates sections of your hard disk drive to act like the RAM memory, but a hard disk drive is very slow. The StringBuilder option looks fine for who use the system as a mono-user, but when you have two or more users reading large files at the same time, you have a problem. Have a look at the following code snippet. You have mentioned Most files will be MB. This claims to read MB in 1.

Original Article. You might be better off to use memory-mapped files handling here.. The memory mapped file support will be around in.

NET 4 when it comes out as release. The link I have given earlier on is a wrapper around the pinvoke to achieve this. You can map the entire file into memory, and view it like a sliding window when scrolling through the file. As the file is loaded, the iterator will return the progress number from 0 to , which you can use to update your progress bar.

Once the loop has finished, the StringBuilder will contain the contents of the text file. Also, because you want text, we can just use BinaryReader to read in characters, which will ensure that your buffers line up correctly when reading any multi-byte characters UTF-8 , UTF , etc.

I know this questions is rather old but I found it the other day and have tested the recommendation for MemoryMappedFile and this is hands down the fastest method. I wanted to make note of this because I searched the web and tested every recommendation I could find to go back and test MemoryMapedFile successfully. Depending on the memory of your machine.

Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Reading and writing very large text files in C Ask Question. Asked 5 years, 7 months ago.

Active 5 years, 7 months ago. Viewed 7k times. Close ; strReader. Message; System. WriteLine excExcept. WriteLine strDataRow ; file. Improve this question. MicroVirus 5, 2 2 gold badges 27 27 silver badges 50 50 bronze badges. Cass Cass 1 1 gold badge 7 7 silver badges 21 21 bronze badges. Opening and closing the output file for every line that you want to write doesn't help — Steve.

Why do you need to count the number of rows? Can you not do count rows and ditch the first one in one hit? Add a comment. Active Oldest Votes. ReadLine ; if strDataLine!