Advertisement
Advertisement


How to read a large file line by line?


Question

I want to read a file line by line, but without completely loading it in memory.

My file is too large to open in memory, and if try to do so I always get out of memory errors.

The file size is 1 GB.

2019/01/16
1
482
1/16/2019 1:32:28 PM


if ($file = fopen("file.txt", "r")) {
    while(!feof($file)) {
        $line = fgets($file);
        # do same stuff with the $line
    }
    fclose($file);
}
2016/12/07

You can use an object oriented interface class for a file - SplFileObject http://php.net/manual/en/splfileobject.fgets.php (PHP 5 >= 5.1.0)

<?php

$file = new SplFileObject("file.txt");

// Loop until we reach the end of the file.
while (!$file->eof()) {
    // Echo one line from the file.
    echo $file->fgets();
}

// Unset the file to call __destruct(), closing the file handle.
$file = null;
2016/01/19

If you're opening a big file, you probably want to use Generators alongside fgets() to avoid loading the whole file into memory:

/**
 * @return Generator
 */
$fileData = function() {
    $file = fopen(__DIR__ . '/file.txt', 'r');

    if (!$file)
        die('file does not exist or cannot be opened');

    while (($line = fgets($file)) !== false) {
        yield $line;
    }

    fclose($file);
};

Use it like this:

foreach ($fileData() as $line) {
    // $line contains current line
}

This way you can process individual file lines inside the foreach().

Note: Generators require >= PHP 5.5

2017/03/27

There is a file() function that returns an array of the lines contained in the file.

foreach(file('myfile.txt') as $line) {
   echo $line. "\n";
}
2015/07/31

Use buffering techniques to read the file.

$filename = "test.txt";
$source_file = fopen( $filename, "r" ) or die("Couldn't open $filename");
while (!feof($source_file)) {
    $buffer = fread($source_file, 4096);  // use a buffer of 4KB
    $buffer = str_replace($old,$new,$buffer);
    ///
}
2012/11/06