Writing large text files  
Author Message
Jim Mace





PostPosted: Visual C# General, Writing large text files Top

I'm trying to create a simple application that loops through all of the tables in a selected database where a stored procedure creates the insert statements of the data contained within each table. The stored procedure returns a result set of all of these statements. Currently, I am looping through each record within the result set and writing it to a text file. The problem that I am having is that the application is taking a very long time to write some of these files. In particular, one of these files is over 115 MB and it takes the application close to 15 minutes to write this file. Is there rather than what I'm doing now (the write file code is below)

SqlCommand cmd = new

SqlDataReader reader =cmd.ExecuteReader ();

StreamWriter sw= null;

FileInfo t = null;

if

{

}//end if

if

{

t = new

}

else

{

t = new

}//end if

sw = t.CreateText();

sw.AutoFlush= true;

//write insert statement

while (reader.Read())

{

sw.WriteLine(reader.GetValue(0));

}//end while

sw.Close();

sw= null;

reader.Close ();

reader = null;

Console.WriteLine("");

Console.WriteLine("The table: " +tableName+ " has been scripted successfully.");

}

catch (Exception ex)

{

Console.WriteLine("");

Console.WriteLine(ex.Message);

}



Visual C#20  
 
 
James Curran





PostPosted: Visual C# General, Writing large text files Top

Much of that can be reduced:


SqlCommand cmd = new
using (SqlDataReader reader =cmd.ExecuteReader ())
{
// Does not fail if directory already exists

// Automatically truncates if file exists.
using (StreamWriter sw= new StreamWriter(
{
sw.AutoFlush= true;
//write insert statement
while (reader.Read())
{
sw.WriteLine(reader.GetValue(0));
}
//end while
}
}
}
catch (Exception ex)
{
Console.WriteLine("");
Console.WriteLine(ex.Message);
}

However, since that doesn't affect the inner loop of the code, that shold have little effect on it's speed.

The one thing I'd suggest is to leave AutoFlush set to false. Basically, setting it to true tell StreamWriter to stop doing things that might speed it up.



 
 
RizwanSharp





PostPosted: Visual C# General, Writing large text files Top

All things seem fine in your code, I'll suggest only only one thing and hope it'll make some differnece in speed of file writing.

Do no set sw.AutoFlush= true; and keep it as default.....

And do a single flush after all things has been written to file just before the Close();

Here:

reader.Flush();

reader.Close ();

I hope this should make a difference.

Best Regards,

Rizwan