Matheus Tavares

Committing without git

Thu, 28 Dec 2023 00:00:00 -0300

Our goal is to create a branch with two commits: the first adding a single README file, and the second changing this file slightly. All of this without running git.

First of all, why?

Chances are if you clicked on this post I don’t have to convince you why anyone would want to do this… ‘cause it’s fun! :) But also, it should help understand some of the main data structures in git, the “git objects”. I personally think understanding how they work and how they relate to one another is deeply valuable and allows for a better experience when using git.

What will you need?

This is a hands-on tutorial, in the sense that you should be able to copy the code snippets and make commits without git on your own machine. We will be doing everything with python (I wanted to do it in Bash, but Bash strings cannot include the null byte and that complicates things quite a bit).

NOTE: although following these steps should produce a valid git commit, it’s advisable not to use this “in production” as git commands will perform a lot of safe checks and special handling that we will skip for the sake of simplicity. If you need to create commits programmatically, check libgit2.

What is a git object, BTW?

Object are immutable units of storage in Git. They are compressed using the DEFLATE algorithm, and referenced by the SHA-1 hash of their contents (though there is a work in progress to support SHA-256). You can see the objects at the .git/objects directory of any git repo; some will be inside two-hex-digits subdirectories (these are called loose object), and some will be collective stored in files under .git/objects/pack (these are called packed objects). The latter is more optimized as it allows to use “deltas” to minimize redundancies when storing similar objects.

Objects are one of the most important structures in Git; they are responsible for storing the different versions of the project files, as well as other metadata like the structure of the directories, authors and dates of each commit, etc. Here is an example containing the four most important git objects and how they interact with one another:

Yes, commits are not diffs, but snapshots of the whole project at a given time (stored in an efficient manner, of course). The diffs are generated on-the-fly when needed, such as when running git show <commit>.

Making a blob

Without further ado, let’s start writing some code. To start, we need a way to make blobs (Binary Large Objects), which are used to store file contents. A blob has the following structure:

blob {content size}{null byte}{content}

In fact, the header part is similar across different objects, so let’s create a write_obj() auxiliary function, which will be responsible for creating and storing a loose object in our git repo:

import hashlib, zlib, os

def write_obj(objtype, content):
    '''
    Writes the loose object to the git object data base and returns
    its sha1 object. The content must be a byte stream.
    '''
    data = f"{objtype} {len(content)}\0".encode() + content
    sha1 = hashlib.sha1(data)
    hex_sha1 = sha1.hexdigest()
    path = f".git/objects/{hex_sha1[:2]}/{hex_sha1[2:]}"
    os.makedirs(os.path.dirname(path), exist_ok=True)
    with open(path, "wb") as f:
        f.write(zlib.compress(data))
    return sha1

Writing a blob now it’s a piece of cake:

def write_blob(content):
    return write_obj("blob", content.encode())

Making a tree

Next, we need a way to describe the directory structure which will contain our README file. That’s the job for a tree object, which is structured as:

tree {content size}{null byte}[list of entries]

Where each entry has the format:

{mode} {filename}{null byte}{sha1 of tree or blob in binary}

We won’t go into to much detail about the file mode, so its sufficient to say that 100644 represents a regular file with no execution permission.

Note that, in git, all the file attributes (i.e. its name and mode) are stored in the directory structure, not the blob. The blob only holds the file contents. “What about subdirectories?” - you may ask. These are represented by trees as well, whose hashes are referenced by the parent tree. Anyway, here is our write_tree() function:

def write_tree(filenames, hashes):
    mode = 100644
    entries = b""
    for name, hash in zip(filenames, hashes):
        entries += f"{mode} {name}\0".encode() + hash
    return write_obj("tree", entries)

Commit it

The last step is to create a commit to encompass our tree object. A commit in git has the following structure, where the “parents” section is optional and may contain as many parents as needed:

tree {hash}
parent hash1
parent hash2
...
author {name} <{email}> {author_date} {author_timezone}
committer {committer} <{email}> {committer_date} {committer_timezone}

{commit message}

Some important things to highlight here: the author and committer date are stored as seconds since the Epoch; the timezone is encoded as an offset from UTC; and yes, the author may be different from the committer. This happens, for example, when you cherry-pick a commit authored by another person. The new commit will have the same author, but you will be set as the committer.

OK, let’s write our function to create a commit:

import time

def write_commit(tree_sha1, parents, author, committer, msg):
    content = f"tree {tree_sha1}\n"
    for parent in parents:
        content += f"parent {parent}\n"
    content += f"author {author} {int(time.time())} {time.strftime('%z')}\n"
    content += f"committer {committer} {int(time.time())} {time.strftime('%z')}\n"
    content += "\n"
    content += msg
    return write_obj("commit", content.encode())

Branches

With all of the above set, we can write a simple function that takes one of our created commits’ hash and creates a branch on it:

def write_branch(name, hash):
    with open(f".git/refs/heads/{name}", "w") as f:
        f.write(hash)

If this seems confusing to you it may be because we usually think of branches as commit chains, but in reality, they are nothing more than labels. The way git retrieves the commit chain (or better yet, the graph) is by traversing the commits through their “parent” pointers, starting from the branch’s tip commit. Furthermore, if we didn’t create a branch after making our commits here, they would be considered dangling objects because they can’t be reached from any reference (branch, tag, HEAD, etc.). So git’s garbage collector would eventually prune them from the repo.

Let’s test!

Finally, let’s wrap it all together:

# First commit
author = "John Doe <john@doe>"
blob1_sha1 = write_blob("This is a simple README file\n")
tree1_sha1 = write_tree(["README"], [blob1_sha1.digest()])
commit1_sha1  = write_commit(tree1_sha1.hexdigest(), [], author, author,
                            "Add the README file")

# Second commit
blob2_sha1 = write_blob("This is a simple README file\nWith one extra line\n")
tree2_sha1 = write_tree(["README"], [blob2_sha1.digest()])
commit2_sha1  = write_commit(tree2_sha1.hexdigest(), [commit1_sha1.hexdigest()],
                             author, author, "Add another line to README")

# Create branch
write_branch("my_branch", commit2_sha1.hexdigest())

And now, let’s validate it with git:

$ git log -p my_branch

commit 28188fd39b658ff830cd063de722e3803561eef2 (my_branch)
Author: John Doe <john@doe>
Date:   Thu Dec 28 08:07:23 2023 -0300

    Add another line to README

diff --git a/README b/README
index a0a40df..fe62de5 100644
--- a/README
+++ b/README
@@ -1 +1,2 @@
 This is a simple README file
+With one extra line

commit a33ef02efcf8616ff65faf746780971e740c31c6
Author: John Doe <john@doe>
Date:   Thu Dec 28 08:07:23 2023 -0300

    Add the README file

diff --git a/README b/README
new file mode 100644
index 0000000..a0a40df
--- /dev/null
+++ b/README
@@ -0,0 +1 @@
+This is a simple README file

Success! :) You can find a condensed script version of this code here: commit.py.

Git: Rewriting History 101

Fri, 08 Jul 2022 00:00:00 -0300

If there is one thing time-travel movies taught us is that trying to fix the past may lead to dangerous consequences. Fortunately for us programmers, fixing our development branches on Git is quite straightforward if you understand how the commands work and what you should or should not do.

Stick to the end for a terminal example :)

Motivation

Let’s start with a simple question: why would you want to rewrite a Git branch? The most common case, in my personal workflow, is to fix a series of commits before publishing/merging them to a public branch or to apply review comments before sending the next patch set iteration.

WARNING: As a rule of thumb, we should avoid rewriting a public branch that is already being used by others! (Read more about it here and here).

If you do so, the other developers who have work based onto your branch will have a hard time fixing the diverged history later. The easiest way to “fix” a commit that is already in a public branch is to make another commit on top. You can even use git revert <hash> to create a new commit that will completely revert the unwanted commit for you. Besides not breaking the workflow of other devs, this has the added benefit of keeping the “real” history of the code, as it has evolved.

On the other hand, there is not much value in keeping the history of changes made to a personal development branch (before being merged upstream) at each review iteration. Furthermore, you may want to commit frequently (even incomplete work), but then rewrite history to organize your commits before sending a Pull Request, for example. With that in mind, I will now discuss about three techniques that allows us to rewrite branch history.

1) git reset

This is by far the easiest way to “rewrite history” in Git. Passing a commit hash to git reset sets the current HEAD to that commit. If HEAD is a branch, it will will effectively move the branch reference, thus rewriting the branch. See an example below:

Note that git reset accepts many options like --hard, --soft, --mixed, etc. See the git reset man page for more information.

2) git commit –amend

With git commit --amend we can actually modify a commit. However, it won’t get you too far, as it only operates on the commit at the tip of the current branch. You can edit both the commit’s contents (adding, removing, or modifying files) and the commit’s metadata (i.e. the message, author, date, etc.). Let’s see a toy example:

# First, let's create a commit:
$ echo a >a
$ git add a
$ git commit -m "Add a" -- a
$ git log -p

commit 67afb26fbc2dd5018608d61803068405bcba4c6c (HEAD -> main)
Author: A U Thor <author@example.com>
Date:   Fri Jul 8 12:04:57 2022 -0300

    Add a

diff --git a/a b/a
new file mode 100644
index 0000000..7898192
--- /dev/null
+++ b/a
@@ -0,0 +1 @@
+a

# Now suppose we want to amend this commit and add another file to it:
$ cp a b
$ git add b
$ git commit --amend
# By default, `git commit --amend` will open your editor to amend the commit
# message. You can supress this behavior with `--no-edit`. However, since we
# are changing the commit's contents, let's modify the message accordingly.

# Let's see the results:
$ git log -p

commit 9754c684d430ef206cbc65141bea20f82f9d51fa (HEAD -> main)
Author: A U Thor <author@example.com>
Date:   Fri Jul 8 12:04:57 2022 -0300

    Add a and b

diff --git a/a b/a
new file mode 100644
index 0000000..7898192
--- /dev/null
+++ b/a
@@ -0,0 +1 @@
+a
diff --git a/b b/b
new file mode 100644
index 0000000..7898192
--- /dev/null
+++ b/b
@@ -0,0 +1 @@
+a

Note that the hash of the commit changed since its data changed. Now a fun side-track experiment:

$ git log --oneline
9754c68 (HEAD -> main) Add a and b

$ git commit --amend --no-edit
$ git log --oneline
e9475a6 (HEAD -> main) Add a and b

We did not change any files, author, or the message this time. So why did the hash change? That’s because git commit --amend automatically updates the commit date (but not the author date). You can see that running git show --format=fuller 9754c68 e9475a6.

3) git rebase -i (a.k.a. interactive rebase)

Finally, let’s get to the fun stuff!

By default, git rebase takes a set of commits and reapply them over a different base commit. It can be used as a “substitute” for git merge, although the end result in the commit graph is not the same (rebase will rewrite history, merge won’t). I won’t go into any further details about this in this post, but I have some visual explanations at this set of slides.

We will focus on a specific rebase operation mode, called “interactive rebase”, which is enabled with the --interactive (or -i) flag. In this mode, git will let you edit the commits before rebasing them. Note that, if you do not specify a different base, you are effectively only editing the commits :)

Time for another example! For the simplicity, I will use a make_commit bash function which receives a single parameter $1, creates a file named $1 containing the string $1, and then commits the file with a message “Add $1”.

# First let's create a few commits
$ make_commit a
$ make_commit b
$ make_commit c
$ make_commit d
$ git log --oneline

83671c8 (HEAD -> main) Add d
c83b71b files: Add c
c467721 files: Add b
f997aea files: Add a

To specify how far back we want to rewrite our branch, we will give git a base commit. You can use a SHA-1 hash or a revision parameter, like HEAD~3, which roughly means “the third ancestor of HEAD”. So, if we want to modify the last two commits from our example above, we can specify the base either as c467721 or HEAD~2.

Now, when you run git rebase -i HEAD~2, git will open a file called “git-rebase-todo” in your configured editor. This file will contain all commits in the range between (but not including) HEAD~2 and HEAD (including). Something like this:

pick c83b71b Add c
pick 83671c8 Add d

# <a bunch of comment lines>

Each [uncommented] line in this file is an action that git will perform (from top to bottom) on top of our base (HEAD~2), in order to reconstruct the branch. The first word of each line is the command. By default, all lines will have the pick command, which will basically apply the commit as-is. But there are many other commands, and their syntax might differ a bit from each other. Fortunately, you don’t have to memorize anything! All of the commands and their syntaxes are displayed as comments in the git-rebase-todo file for our reference.

Back to our example, suppose we want to change the commit message of the commit 83671c8 Add d. To do that, simply replace pick by reword (or r, as the commands can be abbreviated), save the file, and close it. Git will apply the commit and open your editor with the previous message so that you can rewrite it. After you complete, you can save and close the file, and that is it. The rebase is done :)

What about a more complex change? What if we want to add extra changes to a commit? Well, than we can replace pick by edit, making git stop right after applying that commit and giving us the chance to modify it with git commit --amend. You can even make new commits at this point or use something like git cherry-pick. After you are done with the changes, use git rebase --continue.

During an interactive rebase, there may be conflicts between a commit git is trying to apply (marked with pick) and a previous one that you have modified. In case that happens, you have to delete your whole re… NO, no, wait! Don’t panic. You just have to resolve the conflict, mark the files as resolved with git add, and run git rebase --continue :)

At any point in time, you can also abort the whole rebase operation with git rebase --abort or edit the git-rebase-todo file (with the remaining actions to be performed) by running git rebase --edit-todo.

Finally, it is worth noticing that you can combine two (or more) commits with the squash command. Git will meld the said commit with the previous one on the todo list and open an editor so that you can adjust the message of the combined commit as appropriate. Although there is no command to split a commit in two (or more), you can also do that by using edit. When git rebase hits that commit, run git reset HEAD^ to undo the commit, leaving the changes in the working tree. Then add the changes you want in the first commit and run git commit, repeating that until you have all the individual changes in their respective commits.

A complete example

Extra rebase tips

To remove a commit, you can either use the drop command or simply remove the line from the todo file. Note however that the second way is a bit dangerous as you might accidentally remove a line and have a commit dropped. To avoid that, you can set the rebase.missingCommitsCheck with:

$ git config rebase.missingCommitsCheck error # or warn

This will make git rebase abort if a expected line is missing, forcing you to use drop but also avoiding the accidental drop possibility.

Also, note that git rebase -i can also be used for other things besides rewriting history. The exec command (or the analogous -x CLI option) are very useful when you want to run a given command after each commit in a series. For example, if you want to make sure that every commit in your development branch is buildable and passes the automated tests, you could use something like git rebase -i -x 'make && make tests' HEAD~10. If make && make tests fail in any commit, git will stop the rebase and let you fix the bug before continuing.

Finally, if you need to rewrite all commits in a branch, the HEAD~<n> strategy won’t work… But fear not! There is an option for that: --root.

Epilogue

If you want to know more about rewriting history, I really recommend this chapter from the Pro Git book: https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History.

I hope you liked this post and got as excited as I was the first time I learned about git rebase -i. I think this was one of the features that inspire me to learn more about Git and, later, start contributing to the project. Thanks for reading. Happy [and wise] rebasing :)

The "Schrödinger's tree object": is it there or not?

Sun, 05 Jun 2022 00:00:00 -0300

Reading a reddit comment a while ago, I learned about a somewhat “mysterious” git object which can be both present and absent at the same time! Well… not by the same definition of “presence”, but that spoils the fun, right? :P Let’s see this object in more details.

If you want to check whether a given object exists in a git repository, you can use git rev-parse. As the man page says:

--verify
	   Verify that exactly one parameter is provided, and that it can be
	   turned into a raw 20-byte SHA-1 that can be used to access the
	   object database. If so, emit it to the standard output; otherwise,
	   error out.
	   [...]
	   To make sure that $VAR names an existing object of any
           type, git rev-parse "$VAR^{object}" can be used.

Let’s run an example inside the git.git repository:

$ git rev-parse --verify e83c5163316f89bfbde7d9ab23ca2e25604af290^{object}
e83c5163316f89bfbde7d9ab23ca2e25604af290
$ echo $?
0

Ok. What about an nonexistent object?

$ git rev-parse --verify 0000000000000000000000000000000000000000^{object}
fatal: Needed a single revision
$ echo $?
128

Hm, nothing out of the ordinary here. But the fun starts when we look at the “mysterious” SHA1 hash 4b825dc642cb6eb9a060e54bf8d69288fbee4904. If we list all objects in the git.git repository using git cat-file and grep for this particular hash, we get nothing:

$ git cat-file --batch-check="%(objectname)" --batch-all-objects | \
	grep 4b825dc642cb6eb9a060e54bf8d69288fbee4904
$ echo $?
1

However … rev-parse seems to disagree about the object’s presence:

$ git rev-parse --verify 4b825dc642cb6eb9a060e54bf8d69288fbee4904^{object}
4b825dc642cb6eb9a060e54bf8d69288fbee4904
$ echo $?
0

Hmm, what is happening here? Is the object there or not? Let’s go a bit further with a reduced test case:

$ git init /tmp/repo
Initialized empty Git repository in /tmp/repo/.git/

$ git -C /tmp/repo rev-parse --verify 4b825dc642cb6eb9a060e54bf8d69288fbee4904^{object}
4b825dc642cb6eb9a060e54bf8d69288fbee4904
$ echo $?
0

Wait, what? The just-created /tmp/repo repository clearly has no objects:

$ ls /tmp/repo/.git/objects

/tmp/repo/.git/objects
├── info
└── pack

2 directories, 0 files

Is rev-parse broken? Let’s try something else… Running git cat-file to print all object hashes and grep for our target did not produce any result. But cat-file can also be used to print metadata about a given list of hashes. Let’s try that with our hash:

$ echo 4b825dc642cb6eb9a060e54bf8d69288fbee4904 | \
	git -C /tmp/repo cat-file --batch-check='%(objectname) %(objecttype) %(objectsize) %(objectsize:disk)'
4b825dc642cb6eb9a060e54bf8d69288fbee4904 tree 0 0

Hmmmm, so 4b825dc642 is a tree object of size 0, both on disk and decompressed. Nevertheless, we saw that there are no objects on disk… I was intrigued. Is this hardcoded somewhere in git? And if so, why?

My first attempt to “uncover the mystery” was:

$ git -C git.git grep 4b825dc642
git-rebase--preserve-merges.sh:288:             ptree=4b825dc642cb6eb9a060e54bf8d69288fbee4904
t/oid-info/hash-info:16:empty_tree sha1:4b825dc642cb6eb9a060e54bf8d69288fbee4904
t/t0015-hash.sh:26:     grep 4b825dc642cb6eb9a060e54bf8d69288fbee4904 actual

The first match comes from a shell script which uses our mysterious hash (a.k.a. the empty tree hash) as a fallback if a commit does not have a parent. This is done to compare the hashes of a commit’s tree and its parent commit’s tree to decide whether the commit is considered “empty” (i.e. its tree is the same as the parent). The other two matches come from the test suite. Hmm, so no hardcoded value on the actual object reading code? That’s curious… Let’s see what gdb has to show us! Running git rev-parse -e 4b825dc642cb6eb9a060e54bf8d69288fbee4904 through the debugger, we can see the following call chain:

cmd_cat_file()
  cat_one_file()
    repo_has_object_file()
      ...
        do_oid_object_info_extended()
	  find_cached_object()

And at the footer of find_cached_object() we have this code:

	if (oideq(oid, the_hash_algo->empty_tree))
		return &empty_tree;
	return NULL;

Aha! So we have this “empty tree” hash saved somewhere… Well, turns out that my git grep search did not found it because it is not defined in hex format! See:

#define EMPTY_TREE_SHA1_BIN_LITERAL \
	 "\x4b\x82\x5d\xc6\x42\xcb\x6e\xb9\xa0\x60" \
	 "\xe5\x4b\xf8\xd6\x92\x88\xfb\xee\x49\x04"
#define EMPTY_TREE_SHA256_BIN_LITERAL \
	"\x6e\xf1\x9b\x41\x22\x5c\x53\x69\xf1\xc1" \
	"\x04\xd4\x5d\x8d\x85\xef\xa9\xb0\x57\xb5" \
	"\x3b\x14\xb4\xb9\xb9\x39\xdd\x74\xde\xcc" \
	"\x53\x21"

The last thing to “uncover” is: why is this hash value hardcoded? Well, for that we can find the explanation using git blame (or tig blame). The code at find_cached_object() comes from the commit 346245a1bb ("hard-code the empty tree object", 2008-02-13), which says:

commit 346245a1bb6272dd370ba2f7b9bf86d3df5fed9a
Author: Jeff King <peff@peff.net>
Date:   Wed Feb 13 06:25:04 2008 -0500

    hard-code the empty tree object
    
    Now any commands may reference the empty tree object by its
    sha1 (4b825dc642cb6eb9a060e54bf8d69288fbee4904). This is
    useful for showing some diffs, especially for initial
    commits.
    
    Signed-off-by: Jeff King <peff@peff.net>
    Signed-off-by: Junio C Hamano <gitster@pobox.com>

There we have it, mystery uncovered!

Ok, Ok… I definitely over-dramatized this process… But I find it quite interesting to run this kind of analysis! It helps better understand parts of a code base, reproduce bugs, or even find the reason why a certain function (or line of code) was written in a given way. So I decided to document this particular small adventure. I hope you also enjoyed the “Schrödinger’s object” :)

Faster Git checkouts on NFS and SSD with parallelism

Sat, 31 Jul 2021 00:00:00 -0300

Update (2022/07/13): I defended my Master’s dissertation about parallel checkout and was approved in the program :) Here are the dissertation and defense slides:

MSc Dissertation | MSc Defense Slides

June 2021 marked the release of Git 2.32.0, which includes the new “parallel checkout” mode. This feature can speed up some checkout operations by up to 3.6x on SSDs and 4.5x on NFS mounts!

Furthermore, it benefits not only git checkout, but all Git commands that make use of the checkout machinery, including: clone, reset, switch, restore, merge, and others.

I’ve worked on this project with other Git contributors for about an year, and I’m very happy that it is finally available for everyone to try it out! In this post, I’d like to discuss a little bit about the cases in which parallel checkout works the best, and why is that.

TL;DR

Parallel checkout produces the best results for (large) repos on SSDs and NFS mounts.
It is not recommended for small repos and/or repos on HDDs (without prior benchmarking), as it can worsen performance.
Enable parallel checkout by setting the desired number of parallel workers with git config checkout.workers <N>. One means sequential mode (default).

Motivation
Benchmark
- SSDs
- HDDs
- NFS
Windows benchmark
Conclusions
- How to enable
Notes on parallel-ineligible entries
Source code
Acknowledgments

Motivation ↩

Checkout is usually a fast operation when updating a small number of files, but performance can become a problem as the workload grows. This is specially critical for Git users over networked file systems; which, due to the higher I/O latencies, may experience some checkout commands taking up to 50x or even 130x more time than local file systems.

To put it into perspective, a full checkout of the Linux repository (which contains over 70K files), takes around 8 seconds on a local Linux machine with SSD, but it can take 5 to 15 minutes on network file systems. Furthermore, Linux is not even the largest repository versioned through Git: the Chromium repository has about 400K working tree files with a repo size of 36 GiB; and Windows has over 3.5M working tree files, in a repo of 300 GiB.

With that in mind, the goal of this project was to parallelize the working tree update phase of checkout and improve its performance for large workloads, specially over NFS. Note that other important subtasks of checkout, like the tree traversal or index update code, are beyond the scope of this work.

Benchmark ↩

Since the primary goal was to speedup operations with many file creations, I chose to benchmark a command where this is the main bottleneck: a git checkout . execution on an empty working tree of the Linux kernel repository (version 5.12). This requires the creation of over 70 thousand files.

The command was benchmarked on two instances of the Linux kernel repository:

Packed Objects: a Linux clone containing all objects reachable from v5.12 on a single packfile (8.2M objects, totaling 3.5 GiB of real and disk size);
Loose Objects: a shallow clone (i.e. --depth=1) containing only the objects from v5.12 itself, expanded to loose format (~76K objects, totaling 254MiB of real size, or 454MiB of disk size);

Note: the “Loose Objects” benchmark was created mostly for research purposes. This artificial case should not happen much in practice, since Git packs loose objects when they surpass the 6700 threshold (and our repo have 12x that value).

I’ll interleave the benchmark results with some profiling plots (for the sequential checkout). These were generated to better understand which tasks/functions consume the most time during the execution of the benchmarked checkout command in each storage type. The profiling data were collected using the bcc-tools, which provides both on- and off-CPU profilers, allowing us to properly see the time spent on I/O. For simplicity, I’ll only show a summary of the most time-consuming functions, but you can check this page for the full Flamegraphs.

The benchmark was executed on different Linux machines (with SSDs, HDDs, and NFS mounts), and sampled each measurement 15 times to plot the mean runtime with a confidence interval of 95%. Additionally, the machines caches have been cleaned before each sampling.

Machines info
Please check this page for the hardware and software description of the machines used in this benchmark.

SSDs ↩

Starting with the SSDs, we have three machines: Mango and Grenoble both have 4 cores with 2 threads per core (8 logical cores), and Songbird has 6 cores with 2 threads per core (12 logical cores):

The SSD plots show speedups ranging from 2.5x to 3.6x on the packed objects case, and 5.6x to 7.2x on the loose objects case!

On all three machines, we got the best overall results at around 16 to 32 workers for the packed objects, and 64 workers for the loose objects. But notice that we start seeing diminishing returns around 8 workers. Thus, values higher than that may not be worth the additional usage of system resources (like RAM and I/O bandwidth).

Let’s take a look at the most time consuming functions on a sequential checkout in one of these machines:

For the packed case, most of the time is spent on inflate(), which is the zlib function responsible for decompression. This is CPU-bound, and parallelizes quite well with multiple objects. On the loose case, most of the time is spent on filemap_fault(), which is the kernel function responsible for reading file’s contents on a page fault of a memory mapped region (see mmap(2)). Because of its architecture, SSD allows for internal parallelism, which can be better exploited when there are more outstanding requests in the I/O queue. The increased I/O queue depth also allows for better optimizations to be employed by the I/O scheduler (such as request reordering and merging).

HDDs ↩

We will be looking at three machines on the HDD tests: Wall-e and Cicada have 4 logical cores each, and Grenoble has 8 (all with Intel’s Hyper-Threading).

As we can see, parallel checkout was not very effective on the HDDs. All three machines achieved some improvement in the packed objects case, but the overall speedup was too small. The loose case, on the other hand, saw massive performance degradations. Again, let’s take a look at the performance profile for one of these machines:

As we can see, checkout spends almost all its run time reading the objects from the HDD. However, unlike the SSD, HDDs can typically only execute a single operation at each time. Depending on the I/O patterns, the increase in the I/O queue depth *might* allow for some scheduler optimizations that reduce the overall disk-seeking time. If that is not the case, however, the concurrent requests from parallel checkout may end up only further stressing the disk (think about fights for critical resources), and degrading the performance. I suspect that this is what we are seeing in the loose case checkout. The files may be so scattered over the disk that there is not enough opportunity for request merging by the I/O scheduler. As for the packed case, the scheduler should be able to take more advantage of the increased I/O queue depth, as the objects are stored in a single file.

Yes, there is fragmentation – and the more fragmented case, Wall-e, actually saw higher speedups than the less fragmented ones – however, the number of fragments here don’t come even close to the number of discontinuous disk chunks used to store the loose objects.

NFS ↩

I used two setups for the NFS tests: one with machine Cicada as the NFS server and machine Mango as the NFS client (both on LAN connected through 5GHz Wi-Fi), and another one using two AWS EC2 instances and an EBS gp3 volume for storage (which is SSD-based). Let’s start with the EBS one:

On the Cicada NFS setup, the checkout benchmark with the Linux repository was taking too much time (almost an hour for a single execution of the sequential checkout on the SSD and over two hours on the HDD), so I used the Git repository instead. This repo contains about 4k files at v2.32.0. The packed and loose repository versions were set up exactly like the Linux ones. The first ended up with ~310K objects, and the second ended up with ~4K objects. Let’s see the results:

We obviously cannot compare the times from the two plots above as they are using different repositories, hardware, configurations, and etc. But I wanted to see what difference an SSD makes on a parallel checkout over NFS. Fortunately, Cicada also has a small 20 GiB SSD :) Well, it is meant for caching and accelerating the Windows boot, not for general storage. But let’s give it a shot anyway:

On the NFS tests we got the following speedups (respectively for packed and loose case):

4.5x and 5.2x on the EBS gp3 SSD setup with the Linux repository;
2.6x and 3.4x on the Cicada HDD setup with the Git repository;
2.9x and 5.5x on the Cicada SSD setup with the Git repository.

The best results were at around 32 to 64 workers on all three setups. But we start to get diminishing returns from 8 workers, so this seems to be a good value for NFS mounts, as we can achieve good performance without overusing the system’s resources.

Let’s see the profile plot for the NFS from EBS:

Two functions dominate the execution time: open() with 44% of the total runtime, and fstat(), with 33~40%. For open(), the most time consuming calls are the ones creating new files in the working tree, and their runtime is equally divided between two NFS operations: OPEN and SETATTR. Both of them require one round-trip to the server for each call. As for fstat(), the cost majorly comes from having to flush previous write operations (which were locally cached) to the server. This is required in order to update some file attributes, like the last modification time, which must be returned by fstat(). For both open() and fstat(), practically the entire time is spent off-CPU, either sending requests to the NFS server or waiting on its responses. Since NFS servers are typically able to process multiple connections simultaneously, parallel checkout can be an effective way to amortize the network latency and also promote parallelism in server work associated with these operations.

Finally, I also wanted to see what results we could get on single-core machines. To do that, I disabled all cores but one on both the NFS client and server of the Cicada setup (which can be done by writing ‘0’ to the special files /sys/devices/system/cpu/cpu[1-9]*/online):

The results were very similar to the ones above, with the optimal configuration around 8 to 16 workers. This suggests that the performance gain we get from the parallelism on NFS is also applicable for single-core machines.

Windows benchmark ↩

It would be impracticable to repeat this performance tests in every system and machine architecture where Git is used. However, I still wanted to see how parallel checkout behaves on a different operating system, so I ran the local benchmarks on Microsoft Windows as well. It has a large user base and an active development community on Git. Besides, it is not a UNIX-like system (differently then macOS, BSD, and Linux), so it is a good choice to complement the benchmarks we already have on Linux.

Conclusions ↩

From the benchmarks, I would suggest using something around 8 workers on NFS mounts, and perhaps as many workers as the number of logical cores on local SSDs (at least on Linux).

For HDDs, I would not recommend enabling parallel checkout, unless you have already ran some tests to access the performance on your specific machine and Git repo. You can do that, for example, with something like the following:

#!/bin/bash
# Note: expects $PWD to be the git repo you want to run the benchmark at.
# Also, hyperfine (https://github.com/sharkdp/hyperfine) must be installed.


tmpdir="$(mktemp -d ./tmp-worktree-XXXX)" &&
git worktree add "$tmpdir" HEAD &&
(
    cd "$tmpdir" &&
    hyperfine \
        -S /bin/bash \
        -p 'rm -rf * && sync && sudo /sbin/sysctl vm.drop_caches=3' \
        -L WORKERS 1,2,4,8,16,32,64 \
        'git -c checkout.workers={WORKERS} checkout .'
)
git worktree remove -f "$tmpdir"

How to enable ↩

Set the desired number of parallel workers with:

$ git config [--global|--local] checkout.workers <N>

The default value is one, i.e. sequential mode.

You can also change the checkout.thresholdForParallelism configuration, which defines the minimum number of files for which Git should enable parallelism. This avoids the cost of spawning multiple workers and performing inter-process communication when there is not enough workload for parallelism. (The default value is 100, which should be reasonable in most cases.)

Notes on parallel-ineligible entries ↩

Some files are currently checked out sequentially, regardless of the checkout mode configured. These are:

Symbolic links;
Regular files that require external smudge filters (like Git-LFS).

This limitation exists to prevent race conditions and to avoid breaking non-concurrency assumption that external filters might have. (You can read more about this here.) Additionally, all file removals and directory creations are currently performed sequentially.

Source code ↩

The three series of patches that compose this project can be seen at:

Acknowledgments ↩

Thanks to Jeff Hostetler and Nguyễn Thái Ngọc Duy, who I co-developed parallel checkout with, and to all reviewers for dedicating their time and effort to improve the quality of this feature. In particular, special thanks to Christian Couder, Junio Hamano, and Derrick Stolee. Finally, thanks to Amazon for sponsoring me in this project.

Git 2.26: Up to 3.3x faster pattern searches in git-grep

Mon, 06 Apr 2020 00:00:00 -0300

Quick Links: Final Essay | Timing Results | Poster

Intro

Two weeks ago, Git 2.26.0 was released! And among many great changes, I’m very happy to say that you can expect faster multithreaded git-grep searches! This is the project I’ve worked on for about a year, as my Google Summer of Code project and Undergraduate Thesis, at the University of São Paulo.

I’ve been posting about the project development here. But with the 2.26.0 release, the project is finally concluded, and I think it’s a great time to wrap it all up and present the final results. For those interested in knowing more about the development process, the final thesis is also attached in the Additional Resources section and the top of the page.

Motivation and Goal

Git has become the most popular¹ version control system for software development. Being used to manage a large variety of projects, with different magnitudes (in both content and history sizes), it must be built to scale. With this in mind, Git’s grep command – which searches for text patterns in the tracked files – was made parallel in 2010 using a producer-consumer mechanism. However, when operating in Git’s internal object store (e.g. for a search in older revisions), the multithreaded version became slower than the sequential code. For this reason, threads were later disabled in this case.

The main goal of this work was to improve the parallelization of the grep command and re-enable threads for all its use cases. In this process, it was also desired to implement an optimized and secure thread access to the object reading functions, which could be future used to parallelize other sections of the codebase.

Development

The initial phase of the project involved running several tests to determine which sections of the code were the most time-consuming. For this task, gprof and perf were used in conjunction with visualization tools such as gprog2dot and FlameGraph. The results showed that, in some cases, the object decompression routines accounted for up to one-third of git-grep’s total execution time. These routines, despite being thread-safe, had to be serialized due to the surrounding thread-unsafe object reading machinery.

With improvements to the object reading code and to the parallelism of git-grep, it was possible to allow safe parallel access to the decompressing functions. This, in turn, resulted in an acceleration of over 3x, with 8 threads on a 4-core processor w/ hyper-threading. Some of the changes, such as reducing the code inside critical sections, also brought speedups for the working tree searches, as we will see in the next section. In addition, the process of studying git-grep’s code and its call graph allowed us to find and fix some race condition cases and a bug regarding searches in submodules.

Results

The following plots compare git-grep’s execution times before and after the proposed changes. The results presented are all means of 30 executions with a 95% confidence interval. Each plot corresponds to a different machine where the tests were executed (grenoble has an HDD and mango an SSD). Also, note that the original code didn’t allow multiple threads for object store searches, but we enabled them just for comparison. See more info at the Tests Methodology Annex.

In the cached searches, we observed speedups of up to 3.34x over the original code, and almost 5x over the original code with threads re-enabled but without the improvements. Additionally, the working tree searches also got faster with our changes, showing speedups of up to 1.53x.

Additional Resources

Final Essay (and other documents)

Below are some additional materials which describe the development process in much more detail. The first (and most complete) one is the final undergraduate thesis; the second one is the poster presented at the university in an open session; and the third one is the initial proposal (which is considerably outdated, since the project’s main idea changed quite a bit during the development course).

Note: Both the Final Essay and the Poster mention that there was still a race condition case to fix. But, as we later discovered, the code was already protected. To understand how, please check this patch.

Code

Below is the list of code patches originated from this work, and sent to the Git project. All of them have already been incorporated into the upstream Git repository.

Solo patch “grep: fix worktree case in submodules” [version 1]
Merged, part of Git version 2.24.0.

[1/1] grep: fix worktree case in submodules

Patchset “grep: improve threading and fix race conditions” [version 3]
Merged, part of Git version 2.26.0.

Annex: Tests Methodology

The plots presented in the Results section were generated with timings of the command below (adding the --cached option when testing searches in the object store). The regular expression chosen represents a realistic and time-consuming search case.

  $ git --no-pager grep --color=never --extended-regexp \
	--threads=<number of threads> \
	'(static|extern) (int|double) \*'

The Chromium repository² was used as test data, for being a relatively large repo in both content and history size. Also, Chromium’s developers had already reported some difficulties regarding a couple of slow Git commands in their daily usage.

As the problem we were trying to solve is related to I/O, the tests were repeated in two different machines: one with HDD (grenoble) and one with SSD (mango). Both powered by quad-core CPUs with hyper-threading, running a Linux distribution. More information about the machines and tests can be found in the Final Essay.

Footnotes

According to the Stack Overflow Developer Survey Results from 2018 ↩
Downloaded from https://chromium.googlesource.com/chromium/src/ at commit 03ae96f (“Add filters testing at DSF=2”, 04-06-2019). ↩

GSoC Follow Ups

Fri, 18 Oct 2019 00:00:00 -0300

This is a quick follow up post to talk about the status of my project after GSoC has finished.

The Git community was very receptive and I definitely wanted to keep contributing to the project after GSoC ended. Besides, my project wasn’t complete yet and I had some ideas in mind I wanted to try. So it was time for a v2!

I also took some time to write a post talking about how to start contributing to Git based on the experiences I had during this period. I’m very happy that some people who wanted to start contributing reported it was helpful in some way :)

Where were we?

As a brief recap, the idea was to make object reading thread-safe (and with good parallel performance) so that we could re-enable threading in git-grep for the non-worktree case. Part of this goal was already accomplished during GSoC by allowing zlib inflation to be performed in parallel and protecting the other object reading section. However, the code for some of git-grep’s options (namely --textconv and --recurse-submodules) wasn’t fully protected yet, forcing git-grep to run sequentially when these options were given.

Main idea for v2

For this new version, I’ve rewritten the patches almost entirely from scratch. My main idea was to turn the mutex that would protect object reading into a recursive mutex. And then, make use of it at external places that needed to be protected against concurrent object reading operations. In particular, I wanted to use it to protect some of the code concerning git-grep’s --textconv and --recurse-submodules options.

The need for the mutex to be recursive comes from the fact that some of the external places it was used in this patchset also perform object reading behind the curtains (so if it weren’t recursive, the thread would try to double lock the mutex leading to possible errors).

I also realized that protecting only oid_object_info_extended() wouldn’t be enough for the desired usecase at git-grep. So, between other changes, another mutex was added to protect the operations at replace-object.c and some lazy initializers were forced to perform eagerly (before dispatching the worker threads, to avoid races). Namely, these were the .gitmodules file loading and the initialization of packed_git. The grep_submodule() function was also refactored to safelly perform as many parallel submodule operations as possible, aiming to increase performance.

Inspecting call graphs

To make sure we were race-free, before re-enabling threads for all git-grep cases, I decided to generate and inspect some call graphs. Especially, I was looking for paths in git-grep’s call graph that would lead to unprotected functions without acquiring locks. For this task, I tried many tools, such as:

My callpath script
The Cally tool
GNU’s Cflow

But none of them gave me the picture I wanted. The main problem I had with some of the tools was not being able to handle multiple valid functions with the same name, as I described in this issue.

At this point, I got a great help from Giuliano Belinassi, a friend of mine who contributes to GCC. He wrote a patch to make GCC dump the whole call graph for the program being compiled, in dot’s format. (He’s still improving the patch to send it upstream. I’ll try to remember to update this post with the patch link once he sends it.)

Then, I wrote a python script to filter only the paths starting from a group of functions A and ending in another group of functions B (making sure to include all paths, including recursive ones). I also added an option to exclude all paths that contained any function from a group C. With the patched GCC and the filter script, I began the journey of generating and analyzing git-grep’s call graph. This took me a lot longer than I thought it would, but I think it was worth the effort. I focused in searching for paths departing from cmd_grep() or run() (the worker threads’ start routine) and leading to any of the thread-unsafe functions that would also be present in object reading’s call chain (at least, the ones I know to be thread-unsafe).

As an example, I knew that parse_object() wasn’t thread-safe and that it was present at object reading call chains. So I wanted to check for other calls to it outside object reading (as these were already protected). With that in mind, I generated this graph.

I know, it’s a *huge* graph and too hard to be manually analyzed. But starting at cmd_grep()/run() and eliminating the paths we know that are protected (in the dot file, before generating the image), we are left with a smaller subgraph which indicates the currently unprotected (and racy) paths. As an example of unprotected and possibly racy path that can be found in the above graph, we have:

cmd_grep() > grep_objects() > deref_tag() > parse_object()

With this technique, I found other 5 spots in git-grep that were already in race condition, even for the worktree case. This lead to the 3 first patches of this new iteration:

(the other ones can be seen in the link bellow)

Results

The final patches can be seen here. They have been already picked up and are now cooking at pu.

Next Steps

I’m really glad to see git-grep running faster and with full threading support (even when --textconv and --recurse-submodules are used)! Now that the main goal was reached, I want to work on some side issues I found during the process of examining git-grep for this project. The first one is trying to make grep_submodule() stop adding the submodule’s odbs to the in-memory alternates list.

Another problem I stumble across during GSoC is the textconv cache been always written to (and read from) the_repository, even for submodules’ objects. That doesn’t perform well when we git-grep the superprobject and then go down to the subproject to further inspect it. I have told a friend about this issue and he is making progress trying to solve it :) I hope I can pair up with him to help.

I also have to finish writing the monograph for my college capstone project (whose theme is my GSoC project :).

First steps contributing to Git

Mon, 02 Sep 2019 00:00:00 -0300

Git is an amazing tool with worldwide use! Being a tool we use daily, as developers, it’s a real pleasure being able to see how it works internally. And since Git is a free and open source project, we can do that! Even more, we can join the community and contribute to further improve this great software :) That’s what we’ll talk about in this [not so] brief post.

Index

0) Glossary
1) Community
2) Downloading the Source
3) Overview of the Repository
4) Compiling, Installing and Running Tests
5) Submitting Patches
6) Where to Contribute
7) Some Extra Tips
8) Extra References

0) Glossary ↩

There’re some words we’ll see a lot throughout this post, so let’s stop for a minute to recap their meanings. If you’re already comfortable with them, please, feel free to skip this section.

Patch: A patch is a file containing a collection of changes to a given code. When we talk about patches, in this post, we are in fact talking about “source code patches”. This subset comprehends the set of textual patches which can be applied over source code. They are usually a “diff” containing the “additions” and “removals” of code lines. (You may think of a patch as a diff between two commits in a row).
Patchset or series: a set of patches that usually refer to the same thematic. Although each patch has its own set of changes, the patchset usually has a common goal and there may be dependencies between a patch and its “parent”.
Review: a series of comments and suggestions on a patch, to help improve its quality.
Patch version: the patch’s current iteration. It’s very common for a patch not to be merged in its first version. So after getting reviews and fixing what is needed, the author will send a “v2” (version 2). This process may repeat for some iterations (v3, v4, etc.).

1) Community ↩

The Git community communicates mainly through:

The Git mailing list
IRC, in channel #git-devel at freenode.org

Mailing List

In order to subscribe to the mailing list, you have to send an email with no subject to majordomo@vger.kernel.org containing subscribe git at the body. Then wait for a follow-up email from majordomo, which will give the next instructions. Basically, you’ll receive an authentication token that must be sent back to confirm subscription. Finally, if everything goes well, you will receive a welcome message from majordomo :)

Note: All messages to majordomo (and to the mailing list) must be sent in ‘plain text’ mode, without HTML code. In Gmail, you can enable this mode in the tree dots icon when writing an email. Regarding posting style, the preferred method is the inline reply.

What’s Cooking

The maintainer frequently sends a “What’s Cooking” message containing information about each topic branch he’s holding and how they are evolving in the workflow (more on that later). It’s always good to keep an eye on these emails to check on updates. If you are working on a new version of a patch which is marked as “Will be merged into ", you may also reply to Junio asking him not to merge it yet.

Filters

As the volume of messages is quite high, you may also use a combination of filter and label in your mail server to keep the list messages separated from your main inbox. In Gmail, you can create a git label and use the following filter and action options:

Archives and referencing emails

The mailing list also has some archives, which are very handy when looking for specific topics (or conversations that happened before you subscribed). Links to the archive are also quite often used as references when talking about a specific mail thread. Here are some public archives:

When referencing an email, the community usually opts for the first archive. It uses Message-IDs as identifiers in URLs, which makes it quite easy to use as a reference. (As we’ll see latter, you may use --in-reply-to=<Message-ID> of git-format-patch and git-send-email to send a reply to a specific email). Besides that, public-inbox has some really nice filtering options.

Patchwork

There’s a patchwork instance at kernel.org that also tracks all the patches sent to the Git mailing list. You can check it here. It’s a nice way to list patches and their series.

IRC

If you have never used IRC (or perhaps wants to refresh your knowledge of it), FLUSP has a nice tutorial on how to chat on IRC with weechat.

There’re two channels, both at irc.freenode.net:

#git: for questions on how to use Git. There’s a log here.
#git-devel: for Git development discussions. There’s a log here.

Part of the community also gets together at IRC every other Monday at 17:00 UTC for a virtual standup meeting. All are welcome to join in :) To avoid missing the dates, you might want to subscribe to the Git Events calendar.

Calendar

Talking about calendars, we also have the public Git calendar where it’s possible to check the development phases for each Git version.

2) Downloading the Source ↩

Git’s source code is available at https://git.kernel.org/pub/scm/git/git.git/.

There are also some alternative mirrors at:

You may also want to check Junio’s (the maintainer) fork at GitHub. There you’ll not only find the main integration branches but also individual topic branches being cooked: https://github.com/gitster/git/.

To get familiar with the branches workflow, you can read the man gitworkflows page.

Some sections of the system have dedicated maintainers with their own repositories. If you wish to contribute to these sections, your work must be based on their trees. These sections are:

‘git-gui/’, maintained by Pratyush Yadav: https://github.com/prati0100/git-gui/
‘gitk-git/’, maintained by Paul Mackerras: clone from git://ozlabs.org/~paulus/gitk
‘po/’, maintained by Jiang Xin: https://github.com/git-l10n/git-po/. Note that you can also contribute with translation through weblate.

3) Overview of the Repository ↩

Git is mostly written in C, but there are also some perl and shell scripts. You’ll see that each command, in general, has its own command.c, command.perl or command.sh file. However, that’s not always true. For example, the git status’s entry function, cmd_status (another pattern), is at builtin/commit.c.

Tip: Take some time to browse the code and inspect the implementation for some of the commands you use daily. It’s really fun! You can check, for example:

Git’s main function at common-main.c
Some commands’ entry functions (usually cmd_*)
The way Git declares options (the struct option arrays for each command)

The t directory is where the tests reside. We’ll talk more about them later, but it’s important to highlight the t/README file. That’s were the test lib, test structure and execution options are described.

The Documentation directory is a very precious information container. There you will find, for example:

many git-*.txt files, which becomes the Git manpages;
the technical dir, containing nice descriptions of APIs and protocols;
the howto dir, containing tutorial-like articles for both users and contributors.

How-to-contribute documentation

The Documentation dir also holds some very important files on the contributing process. Make sure to read these before sending your first patch.

SubmittingPatches: contains a lot of useful information on the contributing process. Here is the HTML version.
CodingGuidelines: describes the coding style used by the community.
MyFirstContribution: a great tutorial on how to start contributing to Git. It helps better understand the code flow, by walking the reader through the process of creating a new Git command! Here is the HTML version.

4) Compiling, Installing and Running Tests ↩

Compiling

To compile, run make -j<num_threads>, replacing <num_threads> by the desired number of threads.

Note: You may need to install some additional dependencies. But if compilation fails, you can check what you need to install by the error messages.

config.mak

This file is ignored by Git and should be used to enable custom compilation options. It is automatically included by the Makefile.

One thing you may want to do at this file is setting the DEVELOPER knob. This will enable the most important compilation warnings Git community cares about. To do so, simply add DEVELOPER=1 to your config.mak.

The config.mak file is also very useful when you want to compile Git to run it through GDB or Valgrind. For example, you may use:

LDFLAGS += -g3 -O0
CFLAGS += -g3 -O0

Installing

Normally, to install Git in your ~/bin directory, you can simply run make install.

If you want to do a global install, run:

make prefix=/usr all doc info # as yourself
make prefix=/usr install install-doc install-html install-info # as root

Warn: be sure to carefully read the INSTALL file for more information.

Testing

Tests are under the t/ directory, and you can run them by cd-ing to the directory and invoking make, to run all tests, or ./<test_file>.sh to run a specific test file.

When debugging a test you may run it with -v -d -x (check t/README for individual descriptions) and get more information on the test execution. Also, the directory created to run the test (something like trash directory.tXXXX-YYY.sh) won’t be deleted even if there’re no fails so that you can inspect it after the test is finished.

Since Git tests output TAP, it’s possible to run them with any TAP harness such as prove. Please, read t/README for more information on this.

Travis-CI

There’s a .travis.yml file which, in combination with some scripts in ci/, enables Travis-CI testing for Git’s repository. As running all tests can take quite some time and Travis-CI gives support to some additional tests (running on both Linux and macOS, for example), it’s a nice option to test your patches. To set it up, just fork Git’s code in GitHub, sign in to Travis and enable builds for Git. Every time you push changes, a build will be triggered!

5) Submitting Patches ↩

First of all, it’s important to know what branch to base your work on. You should read Documentation/SubmittingPatches to know more about it (and other important information). If you are not used to Git’s branches division and workflow, you might also run git help workflows to read more about it. Also, don’t forget to check Documentation/CodingGuidelines, for the used coding style. This is taken very seriously.

Commit message

Your work should be divided into separate commits for logically separated changes. And for each commit, try to describe the changes with a meaningful message. There’s plenty of information about this at Documentation/SubmittingPatches. This post on how to write good commit messages is a nice reference as well. And finally, it’s also a good practice to run git log --no-merges for some real examples.

Tip: Don’t underestimate writing commit messages. It’s a very important section of the process.

You can use tags in your commit message. The S-o-B (Signed-off-by) is a required one. But you can also use Helped-by, Co-authored-by, Suggested-by, Reported-by, Original-patch-by, and others. Again, take a look on their uses running a git log at the repository. (I also plan to do a separate post about them, soonish)

How to send

After you have completed all of the above steps and verified the patches’ correctness (compiling the code without warnings and successfully running the test suite), it’s time to submit your contribution. As you might have already noticed, the patch submission and revision process happens in the mailing list itself (remember, GitHub and Gitlab didn’t even exist when Git was first released).

Does that mean we need to manually format the patch, copy and paste it into the email client and send it? Well, you can do that. But be careful as some clients may mess up with tabs and/or add HTML tags, corrupting your patch! Therefore, unless you are sure that your email client won’t cause you problems, it’s better to avoid this approach…

So what can we do, then? Hmm, it would be great to have a program that takes a commit from a local repository, formats it and automatically sends it to the list, right? Oh, wait… we have such a program! Git! haha You can use git-send-email to do that. Here’s a tutorial on how to configure and use it. In general, you will probably use something like this:

git send-email \
     --annotate \ # open in editor before sending
     --cover-letter \ # add a cover letter (only for series, i.e. >1 patch)
     --thread --no-chain-reply-to \ # controls mail threading
     --to="git@vger.kernel.org" --cc-cmd="contrib/contacts/git-contacts" \ # to and cc
     -v2 --in-reply-to="<MESSAGE-ID-HERE>" \ # sending a v2 as reply to your v1's cover-letter
     -2 # number of patches

(You can check more about the command added to --cc-cmd in this example at contrib/contacts/git-contacts.txt.)

Alternatively, you may also use GitGitGadget, which makes the connection between Pull Requests and the mailing list :) Here is more information on this.

Reviews

After some days you will probably receive some comments on your patch. You might also want to see if it has already been queued to a topic branch in the maintainer’s fork. (you’re looking for a branch named xy/topic-of-this-patchset where xy is the contributor’s namespace, usually your initials). If that’s the case, keep following the “What’s cooking” messages to see the status of your patch. And reply to the maintainer if you’ll be sending a new version of a patch marked to be merged into next, so that he can postpone merging it.

Sending a new version

If you received reviews, do not forget to thank the reviewers and reply to their comments. It’s a good idea to reply them individually so that reviewers know what you are doing in v2. You may also suggest other ways to address the raised points, but it’s important to be opened to other’s suggestions as well :) Then, you can make the necessary modifications using git rebase -i and editing each commit individually.

Info: In theory, you could add a new commit for each necessary fix in your v1. But since the patches haven’t been merged yet, it’s much better to send a corrected version of them than a series containing a patch with a mistake followed by another one fixing it.

When the new version is ready, you should send it as you did before. This time, though, you should add the v2 flag (you can use git send-email -v2 for that) and send it in reply to your v1’s cover letter. To do that, get the cover letter’s Message-ID in the public-inbox and give it to the --in-reply-to option of git-send-email. This is important as the history of the patchset is kept together. Oh, and for a single patch you may send the new version in reply to the patch itself (as there will be no cover letter).

Tip: Develop each patchset version in a different branch. With that, you can use git diff <branch-v1> <branch-v2> to quickly check your changes and use git-send-email’s --range-diff option to send a version changelog with your patch(es). (Check more about it in git-format-patch’s man page.)

I didn’t get replies

If you didn’t get replies and your patch wasn’t queued yet, be patient and give it a couple more days. The other developers are also working on their tasks and may not have had time to see your patch yet. Also, check the calendar if a new release is approaching. In these periods new features that are not yet ready for the upcoming release are likely to get less attention, as they are not as urgent. However, if that’s not the case and there are still no replies in a few weeks, you may resubmit the patch with the tag “RESEND PATCH” or just incrementing the version and commenting it in the cover letter. But again, be patient :)

Some more tips on patch sending

Here are some extra tips on patch sending. The flags you’ll see refer to git-send-email and git-format-patch:

Comments section: There’s a “comments section” in each patch. The lines added between the three-dash line (---) and the beginning of the diff will be excluded when applying the patch. So you may use this space to add any additional information that you don’t want to be saved in the commit message. (Such as questions, simple comments or changelogs)
RFC and WIP: If your patch (or patchset) is not finished yet but it’s almost there and you want some comments from the community, you may use the [RFC] and/or [WIP] tag. Respectively, they mean “request for comments” and “work in progress”. Usually, we use [RFC PATCH] for a patch pretty close to conclusion and [RFC WIP PATCH] for an even cruder version. (You can set them using --subject-prefix="RFC PATCH", for example)
git-send-email options: git-send-email accepts many options from git-format-patch as well. Thus, when you don’t find a flag in the former’s man page, try looking for it in the latter’s one.
Area prefix: it’s very common to use an “area” prefix in the commits’ titles. For example, if you are factoring out a function at config.c, your commit title may be config: factor out obj reading from write_commit(). (If you’re not sure about the prefix to use, run git log --no-merges <file>, in the file you changed, for some real examples.) Also, do not confuse this with the RFC and WIP tags. While these tags are not kept in the commit message, the area prefix is.
Version changelogs: It’s a good idea to include changelogs between versions so that reviewers may quickly see what has changed. You can add them to the cover letter or in the comments section of each individual patch. The format can be something like this:
```
  Changes since v3:
  - Replaced git_inflate() call for git_inflate_gently()
  - Fixed typo on git_inflate() documentation
```
You can also use --range-diff against the older version.

6) Where to Contribute ↩

For starters, I really like the following advice from Peff:

Scratch your own itch. Surely git doesn’t do something that you wish it did. Or did it faster. Or whatever. Try to dig up past discussions on the list to make sure you’re not doing something that has already been tried and rejected, and then start hacking.

Read the [mailing] list. People will report bugs. Try reproducing them, bisecting them, creating minimal test cases, narrowing the issues down to certain configurations or a certain bit of code, etc. Sometimes that will lead you to propose a solution. Sometimes you’ll just add to the discussion, and then somebody with more familiarity can pick up the topic from there. But you’ll have helped them by doing some of the work, and you’ll have learned more about how git works.

And here are some extra ideas to help you get started:

GSoC Microprojects: Git usually participates in Google Summer of Code. As a preparation/exercise for applicants, the community strongly recommend them to submit a small code-related project as part of their application. You may take a look in the microprojects from the last years to see if there’s something you can contribute to. (Note: the link refers to 2019 microprojects list but feel free to check links from other years as well)
Search for #leftoverbits in the mailing list. Here and here are some examples.
Look for some tags in the code: FIXME, TODO, NEEDSWORK, and BUG (not to be confused with the BUG() macro). You can search them with a git grep NEEDSWORK, for example.
Emphasizing, keep an eye on the mailing list. There’re often discussions on feature requests, bug reports, etc. It’s also a good idea to read other people’s patches to keep up with the overall development process. This might as well ring a bell on other contributions you can work on :) And speaking of mailing lists, you may also want to check the users mailing list once in a while.
Git community don’t have an official issue tracker (as already said, the list also serves this purpose, being used to report and discuss bugs). But there’re some other places where people have been filling and tracking bugs/issues:
- crbug
- GitGitGadget
And if you are working on Git-for-Windows, you can check its issue tracker here. You may also find general Git issues there as well.
Look GSoC/Outreachy project ideas from the past years that weren’t selected. Although they probably require much more work than what’s suitable for a first contribution, you might try selecting subtasks from them. Here is Git’s list of project ideas for GSoC 2019, for example.
Junio’s blog has a list of leftover-bits. (Check if the issue isn’t solved yet. You may also talk about it in the mailing list.)
Finally, you may check these three pages containing project ideas for Git at kernel.org:
- SmallProjectIdeas (and the respective discussion on the mailing list)
- Wishlist
- Janitor
Note: some of these pages may not have been updated recently, so it’s good to check if the issue isn’t solved yet, before trying to do it.

7) Some Extra Tips ↩

Sometimes we get a little lost in Git’s technical terms. A good friend in these moments is man gitglossary!
Following the same idea of the previous item, there’s man gitrepository-layout. It is very handy to better understand the structure of a Git repository.

8) Extra References ↩

Although this post didn’t end up so short, it certainly doesn’t cover all aspects of the contributing process. So here are some great extra readings:

Introduction to Git - talk by Scott Chacon
Pro Git - book by Scott Chacon and Ben Straub
- Section 10, Git Internals, is a spectacular reading for those wishing to know more about objects, references, packfile format, etc.
Understanding Git — Index - post by Zvonimir Spajic
Git for Computer Scientists - post by Tv
Git docs, especially:
- Glossary: explain many technical words.
- Workflows: describe the workflows used at git.git.
- Revisions: an explanation of the extended SHA-1 syntax.
- Repository layout: the structure of repositories created through Git.
- MyFirstContribution: a great tutorial on the contributing process.
- SubmittingPatches: a must-read for everyone who wants to contribute to Git.
- Other doc files, not present online, such as Documentation/CodingGuidelines and Documentation/technical/*

Tip: I recommend you to read, at least SubmittingPatches, CodingGuidelines and MyFirstContribution while working on your first patch.

That’s it for this post! I hope to see you soon in the mailing list and IRC :) (BTW, you can find me as “matheustavares” there).

Contribute to this post :)

If you find any problem with this post and/or want to propose improvements, please, feel free to create an issue or submit a merge request here.

GSoC Final Report

Sat, 24 Aug 2019 00:00:00 -0300

Note: I wrote a small follow-up to this post talking about what I’ve done in my project after GSoC has finished. You can check it here.

So, this has been the final week of GSoC. It passed so fast! Let’s look back, summarizing what was done, the current state of the project and what’s next.

You may also want to check out my weekly posts on the project, which can be found here.

The project

You can check my project proposal here. The central idea behind it was to allow threading to more Git commands by making the pack access code thread-safe. We initially planned to protect the pack access functions before working to parallelize any specific Git command. However, without a use of the pack access API in mind, it was difficult to know which snippets really required thread-safety. In fact, I could potentially be working on snippets that wouldn’t necessarily need to be called in parallel. Also, we weren’t able to fully test and validate the changes (or its real benefits).

So we changed the approach and decided to work on the pack access code from the perspective of git-grep. This command was already parallel, except when grepping the index (with --cached) or given trees. And the reason for that was due to poor performance when the object store was involved, because it had to be handled sequentially. So git-grep was really showing itself as the perfect candidate for the project.

Most of GSoC’s first half, I invested in profiling and code analysis (also in my microproject, as I’ll talk about in the next section). I was trying to validate the hypothesis that allowing parallel object reading would indeed speedup git-grep. Even more, I wanted to know which sections of object reading would be more suitable for parallelization. And that’s when I started to investigate zlib inflation. You can check some of the plots and analysis made in this blog post.

With this patch, I measured the time git-grep spends on git_inflate() alone. And in my test case¹ it accounted for 49.14% of the total execution time! This finding defined which way the project would go from there. From this point, we focused on allowing parallel inflation and making other improvements to git-grep’s thread mechanics.

The patches

As my first patch in the project, I tried to remove function-scope static variables from thread-unsafe pack access functions. We ended up not sending this patch as we changed the project approach, but you can check it here:

sha1-file: convert some static variables to local

For my microproject, I worked improving the internal dir-iterator API and using it in place of readdir/opendir/closedir at builtin/clone.c (to copy or link objects in a local clone). Although I started this patchset before GSoC began, there were 8 iterations and the final version was only sent in the middle of July (it wasn’t so “micro” anymore haha). Here are v8’s patches:

All of the above were already merged into master and were part of Git v2.23.0.

Working at git-grep, I fixed a bug which would make git grep --recurse-submodule always grep the index for each submodule (whilst grepping the worktree should be the default). Here is the final v3:

grep: fix worktree case in submodules

This was merged into master (but not before v2.23.0, so it should be in the next release).

The next patchset was the series where I added internal locks to object reading, releasing it before inflation and reacquiring right after. This way we could have thread-safe object reading with the benefit of parallel inflation. In the same series, I also changed git-grep to use this lock instead of its own, which lead to a 3.00x speedup, as we’ll see later.

These were sent and merged into pu.

The above series re-enabled threading when grepping the index with a good speedup, but it didn’t support some flags: --recurse-submodules and --textconv. This was due to the thread-unsafeness of functions used when these options are enabled. So I kept working on a version to protect them and allow threads in these cases as well. The following patchset is still quite raw and thus, not sent yet. But it’s possible to see the ideas involved:

Note: You can check all the merged patches it the official repository, searching for “Matheus Tavares” in the respective branches. For example, here are the searches in master and pu.

Time Results

To see how well our optimizations to the cached git-grep have gone, I tested it at chromium’s repo⁴, mainly for two reasons:

It has a big object store (around 15GB), which really makes some operations slow
Chromium’s developers already reported some difficulties with slow Git commands⁵

The regexes used for the tests were:

Regex 1: abcd[02]
Regex 2: (static|extern) (int|double) \*

For each of them, the command git --no-pager grep --color=never --threads=<THREADS> <REGEX> HEAD was repeated 30 times after 2 warmup runs and the average execution time was taken. This was done to mitigate system fluctuations. All tests were executed on an i7-7700HQ with 16GB of RAM and SSD. Finally, here are the results:

Threads	Regex 1	Regex 2
1	17.3557s	20.8410s
2	9.7170s	11.2415s
8	6.1723s	6.9378s

This represents a 3.00x speedup, in the best case, which is very nice! But to make sure the optimization also performs well on HDD and/or older machines, the tests were repeated on an AMD Turion 64 X2 TL-62 (dual-core) with 4GB of RAM and HDD (SATA-150, 5400 rpm):

Threads	Regex 1	Regex 2
1	40.3347s	47.6173s
2	27.6547s	35.1797s

And again, we got a quite good time reduction!

What’s next

We already have a good speedup, but the project isn’t finished yet. In this last couple weeks, I’ve been struggling to allow threads when grepping the index with --recurse-submodules or --textconv. The problem is that the function calls involved in these options are not thread-safe yet and they might as well conflict with the now protected object reading functions.

So, after GSoC, I plan to continue this project tackling this issue. I already have some drafts on how to do it. For example the last patchset on The Patches section (which wasn’t sent due to some yet present data races).

However, I’m planning a more incremental process, seeking the best performance gain without changing too much of the code at a time. Certainly, the solution won’t be ideal, but it should bring some improvement in the short term, and be much less risky.

So, what I have in mind is to use a single recursive mutex (which Git already supports for Linux and Windows), to protect both the object reading and submodules’ functions. With this, conflicting operations would be avoided through mutual exclusion and submodule’s functions would still be able to load objects without re-locking errors.

This will hopefully allow us to re-enable threadings with --recurse-submodules getting a good speedup and no data races. The downside is that parallelizable tasks such as object inflation on submodule’s initializations would be serialized. But aiming for incremental changes, that’s an improvement we can think of in the future. As my mentors suggested, I can also write a document at Documentation/technical explaining the parallel git-grep mechanics and how it could be further improved.

Finally, there are some extra possible improvements to tackle at git-grep:

Stop adding subrepo’s object directories to the in-memory alternates list. (This is done at builtin/grep.c::grep_submodule())
Move already thread-safe functions out of git-grep’s critical sections, for better performance. One example is the driver pre-loading at builtin/grep.c::add_work().
Protect lazy initializations and refine the big object reading lock.
Once git-grep already supports threads with --recurse-submodules, do the same for --textconv.
[extra] Refactor the call stack originated at submodule-config.c::config_from_gitmodules() so that submodules don’t need to be added to the in-memory alternates list. This should also bring better performance to object reading operations. I already developed two patches⁶ during GSoC to tackle this issue, but I have to revisit and test them before sending.

Adjacent achievements

Because of GSoC and my work on Git, I also got the opportunity to do some additional related activities:

Callpath

Working on my project, I often needed to check if some thread-unsafe function could be present in a call stack or not. GNU cflow is a good alternative. But it only performs static analysis. So I wrote a simple script called callpath which uses GDB + dot to plot all the paths that lead to a specific function in an execution.

Talk at linuxdev-br

I got to present at the linuxdev-br conference, showing a little bit of Git’s code. Renato and I talked about object-oriented techniques in C using Linux and Git as study cases. I focused mostly on the dir-iterator API (which I worked on as my GSoC’s microproject). Our slides are available here and the lecture recording here. Bellow are some pictures of the talk (click to enlarge):

(Pictures by Eduardo Silva)

Git group at FLUSP

I’m part of FLUSP, a group of students at the University of São Paulo focused on contributing to FLOSS projects. This year, some of us started meeting weekly to exchange Git learnings and tips on how to contribute to it. We’ll hopefully see some patches from the group in the ML soon :)

Thanks

I want to thank my mentors, Christian and Olga, for all their support and always encouraging comments. Also, I want to thank Duy for helping me investigate the possible threading improvements in git-grep. And finally, I thank Junio and the community for the reviews and support.

Being part of GSoC on Git was amazing. I certainly want to keep contributing to Git with whatever I can :)

Footnotes

The test was made running time git --no-pager grep --color=never --threads=1 -E '(static|extern) (int|double) \*' HEAD on chromium’s repo⁴ 5 times and taking the means. The results were:
```
Total time:       10.3599s
Time in inflate:  20.3682s
Percentage:       49.14%
```
↩
Patch by Ævar Arnfjörð. I made some small changes and re-sent. ↩
Patch by Daniel Ferreira. I made some small changes and re-sent. ↩
chromium’s repo at commit 03ae96f (“Add filters testing at DSF=2”, 04-06-2019), after a ‘git gc’ execution. ↩ ↩²
https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/oYe69KzyG_U ↩
The patches removing the add_to_alternates_memory() call from submodule-config.c::config_from_gitmodules():
- config: add repo_config_with_options()
- submodule-config: don’t add subrepo to alternates
↩

GSoC Week 13: Going for a [too] big step

Tue, 20 Aug 2019 00:00:00 -0300

We are approaching the end of GSoC. In fact, this is the final week! A working version of git-grep with parallel inflation was already sent and merged into pu. But threading is still disabled when --recurse-submodules or --textconv are given. So I’ve been working on a v2 to support grepping submodules and try some more optimizations along the way. It didn’t seem so complex at the beginning, but I quickly realized I was just at the tip of the iceberg. This was probably the most intense week in my GSoC. It went like this:

1) Submodule’s initializations

At first, to efficiently grep submodules in parallel I thought I just had to internally protect these four functions at the beginning of grep_submodule:

is_submodule_active()
repo_submodule_init()
repo_read_gitmodules()
add_to_alternates_memory()

Doing so, I would be able to remove the grep_read_mutex around them in favor of the internal object reading locks I added. This would let git grep --recurse-submodules --cached take full advantage of the parallel inflation. (the internal locks are released when inflating.)

1.1) Finding the thread-unsafe spots

The first thing to do was to discover why those functions needed to be in a critical section. A comment right above them give the answer:

/*
 * NEEDSWORK: submodules functions need to be protected because they
 * access the object store via config_from_gitmodules(): the latter
 * uses get_oid() which, for now, relies on the global the_repository
 * object.
 */

But inspecting config_from_gitmodules() I discovered the comment was outdated and the function no longer used get_oid(). However, it now called add_to_alternates_memory(), which adds the subrepo object directory as an in-memory alternate to the global the_repository. As this operation would be performed concurrently with the object reading operations by the worker threads, it could cause data races. (Because it writes to the r->objects->odb list, while oid_object_info_extended()’s call chain may need to iterate through it.)

1.2) Removing 1st `add_to_alternates_memory()`

config_from_gitmodules() was adding the subrepo to the alternates memory in order to call config_with_options(), which needs access to the subrepo’s objects. The call chains I found that reads the subrepo objects are:

config_with_options() > git_config_from_blob_ref() > get_oid()
config_with_options() > git_config_from_blob_ref() > get_oid() > git_config_from_blob_oid() > read_object_file()

So, in order to remove the addition to the alternates list, I added a repo_config_with_options(), which would take an extra repository parameter and pass it down in those chains. Then I made config_from_gitmodules() use this new function.

1.3) Thread-safe initializations?

At this point, I was pretty sure the first 3 submodule operations were already thread-safe without the need of grep_read_mutex or even my internal object reading locks. But there was no way to be sure other than going through each and every function in the call chains. I tried that, but as one would imagine, the call chains were too big for me to manually examine line-by-line…

1.4) Removing 2nd `add_to_alternates_memory()`.

I still had to take care of one more thread-unsafe function: the add_to_alternates_memory() at grep_submodule(). Removing it would not only help in the parallelization but would also solve another NEEDSWORK comment:

/*
 * NEEDSWORK: This adds the submodule's object directory to the list of
 * alternates for the single in-memory object store.  This has some bad
 * consequences for memory (processed objects will never be freed) and
 * performance (this increases the number of pack files git has to pay
 * attention to, to the sum of the number of pack files in all the
 * repositories processed so far).  This can be removed once the object
 * store is no longer global and instead is a member of the repository
 * object.
 */

The call was there so that worker threads could access the subrepo’s objects through the alternates list. Thus, to remove it I had to pass the subrepository reference down to the threads and make them use it explicitly. Unfotunatelly, it seems that this patch introduced some data races I wasn’t able to find until this moment :(

2) Other submodule’s data races

I also noticed some other operations on grep_submodule() which weren’t thread-safe:

The call to parse_object_or_die(), when grepping trees, is outside a lock. This function may load objects internally, write to r->parsed_objects_obj_hash and also call lookup_replace_object(), which is thread-unsafe. So calls to it must be done in a critical section.
To clean the memory allocated for the subrepository, repo_clear(&subrepo) was being called right before returning from grep_submodule(). However, by the time this function is called, there’s no guarantee that the worker threads have already finished working on the subrepo. To properly avoid that, we would need to implement a kind of mapping which tells, for each repo, how many objects are yet to be processed. As this would be somehow laborious, I just skipped the memory cleaning for now, to implement a testing version.

3) Thread-safe code inside critical section

I noticed what it seemed to be an already thread-safe code inside add_work()’s critical section. So I moved it out, which lead to a minor but noticeable speedup:

-static void add_work(struct grep_opt *opt, const struct grep_source *gs)
+static void add_work(struct grep_opt *opt, struct grep_source *gs)
 {
+       if (opt->binary != GREP_BINARY_TEXT)
+               grep_source_load_driver(gs, opt->repo->index);
+
        grep_lock();

        while ((todo_end+1) % ARRAY_SIZE(todo) == todo_done) {
                pthread_cond_wait(&cond_write, &grep_mutex);
        }

        todo[todo_end].source = *gs;
-       if (opt->binary != GREP_BINARY_TEXT)
-               grep_source_load_driver(&todo[todo_end].source,
-                                       opt->repo->index);

4) Move the `obj_read_mutex` down

I didn’t mention it before, but some of those four submodule operations use object reading function such as oid_object_info_extended(), which is not thread-safe. To overcome this, I decided to move v1’s obj_read_mutex from read_object_file_extended() down to oid_object_info_extended(). This allowed even more functions to be called in parallel, but some extra locks were needed.

5) Replace `grep_read_mutex` by internal locks

With all the above done, it was time to remove the grep_read_mutex and enable the internal object reading locks. This moment, I also re-enabled threads in the cached grep with --recurse-submodules. Unfortunatelly, the result wasn’t the best. I tried a lot of changes to this main idea, but all of them ended up in a data race, Segmentation Fault, failure to read subrepos’ objects, or performance drop. Each of these alternative versions are in my local branches, which I thought wouldn’t be worthy uploading to share, but here is the “main” one.

Next steps

I believe my idea for v2 turned out to be a very big step and I just got lost in it. So my current idea is to take a step back and go slower. With that in mind, I might end up not finishing this version during GSoC. But I thinkg that’s OK as we already have v1 sent and with good speedup. And I can keep working on this after GSoC, anyway :)

So, instead of making more tweaks to my v2 series, I sat down and analysed the code trying to list what are the real open issues (and how we might solve them):

add_to_alternates_memory(): althought this function cannot be called in parallel, the case in which git-grep uses it, may not be problematic, if:
- prepare_alt_odb() is called at repositories’ initalization (at cmd_grep() for the_repository and grep_submodule() for subrepos). This way, we force eager initialization and avoid data races with the worker threads in the initialization.
- The ent->next = NULL line is moved up in link_alt_odb_entry(). Then, the worker threads may iterate through r->objects->odb without possibly falling on an unmaped memory region. (That’s hacky, but should work).
parse_object_or_die(): again, this should not be a problem in our case, if:
- The object reading operations are properly protected internally.
- prepare_replace_object() is called at repositories’ initialization (as before).
Submodule initialization functions (at grep_submodule): This is the critical part! I re-investigated them and found out is_submodule_active(), for example, has repack_packed_git() in its call chain. This could clearly generate data races with the workers. And as we are working with “indermediate level” locks, I really don’t know yet how to protect these functions without going lower or higher in our “protection layer”. On the other hand, submodule_from_path() has the same problem and it is outside the critical section… So maybe we can ignore those data races which are too improbable?

Regarding the commits I already have, I plan to:

Remove the submodule-config: don't add subrepo to alternates from the series. It is no longer necessary here, so it might be sent separately.
Remove grep: don't add submodules to the alternates list and hold it in standby. It currently brings data races that must be solved if I’m going to send it in the future.
Refactor the patch adding the object reading locks. I plan to protect only oid_object_info_extended() and not read_object_file_extended(). This won’t work for the general case (which I’ll make explicit), but should work in our case as we’ll be forcing eager initialization for alternates and replace maps in all repos.
The grep: move driver pre-load out of critical section should be OK if we have the object reading locks.
The WIP: grep: remove racy call to repo_clear() must be properly finished.

And that’s the plan! Well, I really hope I’m not being too flustered again, and proposing bigger steps than what I can take…

Main difficulties

Main difficulties this week were:

Debugging data races: I used a combination of GDB, ThreadSanitizer and valgring (helgrind and memcheck). But sometimes the racy spot is to occasional to be caught.
Investigating deep call chains: To make a low-level “thread-protection layer” is hard. I needed to go deep in git-grep call stacks to make sure some functions were safe to be called unlocked. And a couple times I was still unsure.

I also spent some time thinking how I should properly deal with:

Lazy initializations: They are great in big codebases as they mitigate the need to keep track of all initializations in the beginning of an execution. But unfortunatelly, they are difficult to handle when threaded, as two threads can try to initialize a resource at the same time. (For example, take a look at prepare_replace_object() and prepare_alt_odb(), which are called by many functions.)
Static function-scope variables: Some functions such as oid_to_hex() return a static buffer. This is a good convenience for the callees and it avoids memory leaks, but it makes the function thread-unsafe.

GSoC Week 12: Simplified version of parallel inflation

Tue, 13 Aug 2019 00:00:00 -0300

Patchset sent, with some limitations

Last week we had a prototype version of the parallel inflation patchset working without race conditions. Even so, it still had some major issues. For example:

use_pack() needed an aditional provisory mutex to properly work in parallel;
--textconv would produce considerable performance drops; and
the code was a bit unorganized with some probably redundant locks.

These items would still take some time to fix. So my mentors and I thought it would be better to send a simplified version, for now, with the already stable improvements. And I could, then, keep working on a more complete version to be sent latter. This is good because is more incremental and, also, we can get community feedback sooner :)

So, this week, I’ve been working on this simplified version. It already brings good speedup to cached git-grep, but the optimization is disabled when --textconv or --recurse-submodules are used. A section was added to Documentation/git-grep.txt to better explain these details. I also repeated the performance tests I’ve been running, this time on an older machine. The final version can be seen here.

grep: don’t add submodules to alternates

While working at git-grep’s critical sections, I came to the following commentary at grep_submodules() (builtin/grep.c):

/*
* NEEDSWORK: This adds the submodule's object directory to the list of
* alternates for the single in-memory object store.  This has some bad
* consequences for memory (processed objects will never be freed) and
* performance (this increases the number of pack files git has to pay
* attention to, to the sum of the number of pack files in all the
* repositories processed so far).  This can be removed once the object
* store is no longer global and instead is a member of the repository
* object.
*/

Since the object store is now a member of the struct repository, I thought it should be possible to tackle this NEEDSWORK as the commentary suggests. I also thought this modification could help me in the process of supporting threads with --recurse-submodules, as it would simplify a bit grep_submodules()’s critical section. So I worked on a patch to make git-grep stop adding submodules to the list of alternates, as you can check here. The patch already works, but some refactoring is needed.

I ran some tests on a “semi-artificial” repository (linux repo with Git as a submodule), but the patched git-grep didn’t showed a significant time drop. I wonder if I need a bigger repository and/or submodule to really see the differences or if, unfortunatelly, this modification ended up not being as effective…

Next steps

My plan for this week is to keep working on a patch on top of the series I sent earlier to allow threading with --recurse-submodules and --textconv. For now, I’ve been focusing on --recurse-submodules, as I think I know more of its code.

One way to proceed towards our goal would be to move the obj_read_mutex down from read_object_file_extended() to oid_object_info_extended(). With this, we could potentially avoid the race conditions between the threads’ calls to read_object_file_extended() and parse_object_or_die() (which calls oid_object_info_extended()).

Nevertheless, I recently discovered that, just by enabling threads when cached (without my patchset), --recurse-submodules already results in race conditions. The problem seems to envolve the userdiff calls as well. I didn’t get to investigate this enought yet, but that should be one of my next tasks.

Finally, I also want to test the “don’t add submodules to alternates” patch, somehow, to see if it is worthy. But I don’t know how yet.

Matheus Tavares

Committing without git

First of all, why?

What will you need?

What is a git object, BTW?

Making a blob

Making a tree

Commit it

Branches

Let’s test!

Further reading

Git: Rewriting History 101

Motivation

1) git reset

2) git commit –amend

3) git rebase -i (a.k.a. interactive rebase)

A complete example

Extra rebase tips

Epilogue

The "Schrödinger's tree object": is it there or not?

Faster Git checkouts on NFS and SSD with parallelism

TL;DR

Table of Contents

Motivation ↩

Benchmark ↩

SSDs ↩

HDDs ↩

NFS ↩

Windows benchmark ↩

Conclusions ↩

How to enable ↩

Notes on parallel-ineligible entries ↩

Source code ↩

Acknowledgments ↩

Git 2.26: Up to 3.3x faster pattern searches in git-grep

Intro

Motivation and Goal

Development

Results

Additional Resources

Final Essay (and other documents)

Code

Annex: Tests Methodology

Footnotes

GSoC Follow Ups

Where were we?

Main idea for v2

Inspecting call graphs

Results

Next Steps

First steps contributing to Git

Index

0) Glossary ↩

1) Community ↩

Mailing List

What’s Cooking

Filters

Archives and referencing emails

Patchwork

IRC

Calendar

2) Downloading the Source ↩

3) Overview of the Repository ↩

How-to-contribute documentation

4) Compiling, Installing and Running Tests ↩

Compiling

config.mak

Installing

Testing

Travis-CI

5) Submitting Patches ↩

Commit message

How to send

Reviews

Sending a new version

I didn’t get replies

Some more tips on patch sending

6) Where to Contribute ↩

7) Some Extra Tips ↩

8) Extra References ↩

1.2) Removing 1st `add_to_alternates_memory()`

1.4) Removing 2nd `add_to_alternates_memory()`.

4) Move the `obj_read_mutex` down

5) Replace `grep_read_mutex` by internal locks