Focus on IT Recommend

Home > history - Line endings messed up in Git - how to track changes from another branch after a huge line ending fix?

history - Line endings messed up in Git - how to track changes from another branch after a huge line ending fix?



We are working with a 3rd party PHP engine that gets regular updates. The releases are kept on a separate branch in git, and our fork is the master branch.

This way we'll be able to apply patches to our fork from the new releases of the engine.

My problem is, after many commits to our branch, I realized that the initial import of the engine was done with CRLF line endings.

I converted every file to LF, but this made a huge commit, with 100k lines removed and 100k lines added, which obviously breaks what we intended to do: easily merge in patches from the factory releases of that 3rd party engine.

What whould I do know? How can I fix this? I already have hundreds of commits on our fork.

What would be good is to somehow do a line endings fix commit after the initial import and before branching our own fork, and removing that huge line ending commit later in history.

However I have no idea how to do this in Git.


git history rewrite branch newline
  this question
asked Jun 18 '09 at 10:37 keo 2,915 2 10 11


5 Answers


I finally managed to solve it.

The answer is:

git filter-branch --tree-filter '~/Scripts/' -- --all contains:

find . -type f -a \( -name '*.tpl' -o -name '*.php' -o -name '*.js' -o -name '*.css' -o -name '*.sh' -o -name '*.txt' -iname '*.html' \) | xargs fromdos

After all line endings were fixed in all trees in all commits, I did an interactive rebase and removed all commits that were fixing line endings.

Now my repo is clean and fresh, ready to be pushed :)

Note to visitors: do not do this if your repo has been pushed / cloned because it will mess things up badly!

  this answer
edited Dec 9 '12 at 20:57 Nick 6,369 2 20 36 answered Jun 29 '09 at 22:08 keo 2,915 2 10 11 1   This is awesome! However, your script has issues with spaces in the filenames. I did something like this instead: find . -type f -print0 -a ( -name ' .[hc]' -o -name '.p[yl]' ) | xargs -0 fromdos –  Enno Mar 6 '10 at 5:12      Agree that this is awesome. One Q, though: The approach here explicitly specifies which file extensions to strip CRs from. I'd like to know if it's possible to instead strip CRs from all and only those fixes that git's text-detection heuristics would consider text. (Git's heuristics look at the actual file content, not filenames.) At first glance anyway, that seems like it would play even better with core.autocrlf=true. –  Chris Mar 29 '12 at 19:09      Not quite what I was asking, but gives an approach that uses the unix "file" command to try to distinguish text vs non-text files. –  Chris Mar 29 '12 at 19:41      The answers to How to determine if Git handles a file as binary or as text? do give some creative ways you might be able to do what I asked about. –  Chris Mar 30 '12 at 16:50      There is a missing -o operand between the .txt and .html filters. –  nextgentech Feb 1 '13 at 4:07  |  show more comment

Going forward, avoid this problem with the core.autocrlf setting, documented in git config --help:


If true, makes git convert CRLF at the end of lines in text files to LF when reading from the filesystem, and convert in reverse when writing to the filesystem. The variable can be set to input, in which case the conversion happens only while reading from the filesystem but files are written out with LF at the end of lines. A file is considered "text" (i.e. be subjected to the autocrlf mechanism) based on the file's crlf attribute, or if crlf is unspecified, based on the file's contents. See gitattributes.

  this answer
answered Jan 8 '10 at 18:25 Greg Bacon 80.6k 18 150 201      thanks for the tip! we were using this feature but was behaving really buggy then (maybe those bugs were corrected by now) see my next answer to see how we avoided this. –  keo Jan 9 '10 at 11:10


Did you look at git rebase?

You will need to re-base the history of your repository, as follows:

  • commit the line terminator fixes
  • start the rebase
  • leave the third-party import commit first
  • apply the line terminator fixes
  • apply your other patches

What you do need to understand though is that this will break all downstream repositories - those that are cloned from your parent repo. Ideally you will start from scratch with those.

Update: sample usage:

target=`git rev-list --max-count=3 HEAD | tail -n1`
get rebase -i $target

Will start a rebase session for the last 3 commits.

  this answer
edited Jun 18 '09 at 10:53 answered Jun 18 '09 at 10:41 Robert Munteanu 44.2k 24 157 242      "commit the line terminator fixes": I don't quite understand this, as HEAD is at the most recent commit, and LF conversion is a commit back in history. –  keo Jun 18 '09 at 10:49      fortunately no one cloned this repo yet. –  keo Jun 18 '09 at 10:49      there is one thing that's not still clear - if I do a rebase, a few patches will have CRLF in them - as they were committed when the files were in crlf - how do I deal with that? –  keo Jun 25 '09 at 15:12


we are avoiding this problem in the future with:

1) everyone uses an editor which strips trailing whitespaces, and we save all files with LF.

2) if 1) fails (it can - someone accidentally saves it in CRLF for whatever reason) we have a pre-commit script that checks for CRLF chars:

# An example hook script to verify what is about to be committed.
# Called by git-commit with no arguments.  The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
# To enable this hook, rename this file to "pre-commit" and set executable bit

# original by Junio C Hamano

# modified by Barnabas Debreceni to disallow CR characters in commits

if git rev-parse --verify HEAD 2>/dev/null
    # Initial commit: diff against an empty tree object


for FILE in `git diff-index --cached $against`
    fhash=`echo $FILE | cut -d' ' -f4`
    fname=`echo $FILE | cut -f2`

    if git show $fhash | grep -EUIlq $'\r$'
        echo $fname contains CRLF characters

if [ $crlf -eq 1 ]
    echo Some files have CRLF line endings. Please fix it to be LF and try committing again.
    exit 1

exec git diff-index --check --cached $against --

This script uses GNU grep, and works on Mac OS X, however it should be tested before use on other platforms (we had problems with Cygwin and BSD grep)

3) In case we find any whitespace errors, we use the following script on erroneous files:

#!/usr/bin/env php

    // Remove various whitespace errors and convert to LF from CRLF line endings
    // written by Barnabas Debreceni
    // licensed under the terms of WFTPL (

    // handle no args
    if( $argc <2 ) die( "nothing to do" );

    // blacklist

    $bl = array( 'smarty' . DIRECTORY_SEPARATOR . 'templates_c' . DIRECTORY_SEPARATOR . '.*' );

    // whitelist

    $wl = array(    '\.tpl', '\.php', '\.inc', '\.js', '\.css', '\.sh', '\.html', '\.txt', '\.htc', '\.afm',
                    '\.cfm', '\.cfc', '\.asp', '\.aspx', '\.ascx' ,'\.lasso', '\.py', '\.afp', '\.xml',
                    '\.htm', '\.sql', '\.as', '\.mxml', '\.ini', '\.yaml', '\.yml'  );

    // remove $argv[0]
    array_shift( $argv );

    // make file list
    $files = getFileList( $argv );

    // sort files
    sort( $files );

    // filter them for blacklist and whitelist entries

    $filtered = preg_grep( '#(' . implode( '|', $wl ) . ')$#', $files );
    $filtered = preg_grep( '#(' . implode( '|', $bl ) . ')$#', $filtered, PREG_GREP_INVERT );

    // fix whitespace errors
    fix_whitespace_errors( $filtered );


    // whitespace error fixer
    function fix_whitespace_errors( $files ) {
        foreach( $files as $file ) {

            // read in file
            $rawlines = file_get_contents( $file );

            // remove \r
            $lines = preg_replace( "/(\r\n)|(\n\r)/m", "\n", $rawlines );
            $lines = preg_replace( "/\r/m", "\n", $lines );

            // remove spaces from before tabs
            $lines = preg_replace( "/\040+\t/m", "\t", $lines );

            // remove spaces from line endings
            $lines = preg_replace( "/[\040\t]+$/m", "", $lines );

            // remove tabs from line endings
            $lines = preg_replace( "/\t+$/m", "", $lines );

            // remove EOF newlines
            $lines = preg_replace( "/\n+$/", "", $lines );

            // write file if changed and set old permissions
            if( strlen( $lines ) != strlen( $rawlines )){

                $perms = fileperms( $file );

                // Uncomment to save original files

                //rename( $file, $file.".old" );
                file_put_contents( $file, $lines);
                chmod( $file, $perms );
                echo "${file}: FIXED\n";
            } else {
                echo "${file}: unchanged\n";


    // get file list from argument array
    function getFileList( $argv ) {
        $files = array();
        foreach( $argv as $arg ) {
          // is a direcrtory
            if( is_dir( $arg ) )  {
                $files = array_merge( $files, getDirectoryTree( $arg ) );
            // is a file
            if( is_file( $arg ) ) {
                $files[] = $arg;
        return $files;

    // recursively scan directory
    function getDirectoryTree( $outerDir ){
        $outerDir = preg_replace( ':' . DIRECTORY_SEPARATOR . '$:', '', $outerDir );
        $dirs = array_diff( scandir( $outerDir ), array( ".", ".." ) );
        $dir_array = array();
        foreach( $dirs as $d ){
            if( is_dir( $outerDir . DIRECTORY_SEPARATOR . $d ) ) {
                $otherdir = getDirectoryTree( $outerDir . DIRECTORY_SEPARATOR . $d );
                $dir_array = array_merge( $dir_array, $otherdir );
            else $dir_array[] = $outerDir . DIRECTORY_SEPARATOR . $d;
        return $dir_array;

  this answer
answered Jan 9 '10 at 11:25 keo 2,915 2 10 11


One solution (not necessarily the best one) would be to use git-filter-branch to rewrite history to always use correct line endings. This should be better solution that interactive rebase, at least for larger number of commits; also it might be easier to deal with merges using git-filter-branch.

That is of course assuming that history was not published (repository was not cloned).

  this answer
edited Oct 10 '16 at 10:06 Eliran Malka 10.8k 4 51 79 answered Jun 18 '09 at 12:01 Jakub Narębski 161k 44 182 220


------splitte line----------------------------