[[Main Page]] -> [[Documentation]] -> [[Software structure]]

This is a brainstorming document for the general software structure of the pCT project and the operation of the full pipeline.

== General guidelines ==
* support multiple programming languages
* open for the programming tools best suited for a specific purpose and development group
* use common/open source packages where ever possible
* every component comes with a unit test

== Software Modules ==
[[File:Software-task-sequence.png]]

=== Main data path ===
* Simulation
** Beam Simulation
** Phantom Simulation
** Detector Propagation
* Detector Response Simulation
* Readout Simulation
* Readout and Raw Data reconstruction
** pct-online
* Detector Reconstruction
** Clustering and Tracking
* Phantom reconstruction
* Imaging

[[File:SignalChain.png|border|1000px]]

=== Utilities ===
* Visualization
* Control
* Monitoring

== Contact information of maintainers/experts for each software module ==

== Build system and package management ==

== Common software modules ==
=== Data model ===
=== Common IO ===
=== IPC ===
=== Logging ===
=== Control ===

Software structure

2021-03-17T16:13:20Z

Mri083: /* Build system and package management */

[[Main Page]] -> [[Documentation]] -> [[Software structure]]

This is a brainstorming document for the general software structure of the pCT project and the operation of the full pipeline.

== General guidelines ==
* support multiple programming languages
* open for the programming tools best suited for a specific purpose and development group
* use common/open source packages where ever possible
* every component comes with a unit test

== Software Modules ==
[[File:Software-task-sequence.png]]

=== Main data path ===

=== Utilities ===
* Visualization
* Control
* Monitoring

== Contact information of maintainers/experts for each software module ==

== Build system and package management ==

== Common software modules ==
=== Data model ===
=== Common IO ===
=== IPC ===
=== Logging ===
=== Control ===

Software structure

2021-03-17T13:38:54Z

Mri083: /* Software Modules */

File:Software-task-sequence.png

2021-03-17T13:38:16Z

Mri083: Sketch of the data flow through the different software modules of the pCT project.

Sketch of the data flow through the different software modules of the pCT project.

2021-03-04T13:52:56Z

Mri083: Adding instructions how to mirror a repository

GitLab best practice

2021-02-23T12:40:33Z

Mri083: /* Git commits */

[[Main Page]] -> [[Documentation]] -> [[GitLab best practice]]

This page is summarizing guidelines how to use Gitlab within the project. Some general information how to work with GitLab can be found under the following
external links:
* [https://about.gitlab.com/2016/03/08/gitlab-tutorial-its-all-connected/ Tutorial: It's all connected in GitLab]
* [https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/ GitLab workflow overview]
* [https://about.gitlab.com/2014/09/29/gitlab-flow/ What is GitLab Flow?]

GitLab is a web service for '''git''', the ''stupid content tracker'' (according to its [https://manpages.debian.org/stretch/git-man/git.1.en.html man page]).
Git implements distributed repositories, git itself does not even make an assumption about a master repository. It is however good to have such a master repository and dedicated strategy how to contribute to it.
This is, where GitLab enters the scene. The web service is hosting remote repositories and implements procedures for synchronization and content merging.

In the pCT project we have master repositories for different sub-projects. Check the repositories in the [https://git.app.uib.no/pct GitLab pCT group].

In this description, we are using the [https://git.app.uib.no/pct/pct-online pct-online] repository as an example.

== Terminology ==
When working with git/GitLab, you will meet a few terms:

* '''branch''': the tip of a sequence of commits, branches share a common history and develop independently from one point in the history
* '''merge''': two branches are brought together and the result commited with a new commit
* '''rebase''': the history of a branch is applied to another branch resulting into a new history containing the commits of both branches
* '''fork''': repository copy in a user space of GitLab service
* '''clone''': local working copy on a machine

== Default branch structure ==
The main repository has the following branches:
* '''production''': the latest production code, in this branch we have release tags
* '''master''': the latest stable release
* '''dev''': the development branch

Note that we are currently only using the '''dev''' branch. The '''production''' and '''master''' branches will be populated while we are progressing with the development.

In addition to those main branches there can be ''feature'' branches where development happens detached from the main branches. A ''feature'' branch is based on the ''dev'' branch and has a limited lifetime.

== Roles ==
There are two very different use cases, the
# '''User role''': A ''user'' wants to download the code, compile it and use it.
# '''Developer role''': A ''developer'' contributes to the development.

As a '''User''', you can download an archive from the GitLab web interface, e. g. [https://git.app.uib.no/pct/pct-online pct-online], by clicking the download symbol like in this screen shot.

[[File:Pct-online-download-button.png]]

However, even if you think you are fitting best into the user role, you might end up faster than you expect coding and changing the project files. Having content management in place right from the beginning is thus a good idea and it is recommended to use the code by creating a '''clone''' of the repository.

[[File:Pct-online-clone-button.png]]

See [[Gitlab Developer FAQ#Create repository clone]] for instructions.

== The developer fork ==
The ''developer fork'' is the main feature in Gitlab's/Github's concept of distributed development, code sharing and code review.

=== Create developer fork ===
The fork can be created from the GitLab web interface by simply clicking on the '''Fork'''-button, see [[GitLab Developer FAQ#How do I create a project fork?]] for details.

To make the development within the project more coherent, you should follow a couple of [[GitLab Developer FAQ#Which rules should I obey for working with the fork? | rules under this link]].

'''Note:''' If the master repository is not public, also your fork won't be public and you have to allow your colleagues to access it, see in the rules above.

=== Synchronizing developer fork with main repository ===
'''This should be done regularly!'''

There is unfortunately no simple synchronization feature for forks within Gitlab (from Gitlab Enterprise Edition 8.2 ''repository mirroring'' is provided). Synchronization is done in a local clone or via merge reqursts.

In fact, the matter is not so easy in itself as there can be conflicting developments in the master repository and your development fork.

The primary goal of the update is to move the head of a branch in the development fork forward in the sequence of commits, from one point in the commit history to a another one. If the branch to be updated has already new commits, meaning the commit histories have diverged, all these commits should be applied to the new point in the history. Its important to note, that no additional commits are created in this process. This procedure is called ''rebase'' as we put the commits or a branch position to a ''new base''.

==== Update fork using local clone ====
A sample sequence for updating a fork of the pct-online repository, assuming the local clone is set up already (see [[GitLab Developer FAQ#How can I make a clone of a project in my fork?]]).
# Fetch branch from the master repository
#:<pre> git fetch git@git.app.uib.no:pct/pct-online.git dev</pre>
#:or if you have configured the upstream
#:<pre> git fetch upstream dev</pre>
# Rebase local branch
#:<pre> git rebase FETCH_HEAD dev</pre>
# Push to fork
#:<pre> git push origin dev</pre>

==== Update fork using merge request ====
Another way of updating the developer fork is through a reverse merge request from the master repository to the developer fork, ''reverse'' in the sense that the flow of contributions is usually in the opposite direction.

'''Note:''' we only want fast-forward merges in this case, this can not be done if the source branch is not a direct continuation of the target branch, i.e. new commits are present. Do the rebase in the local clone in such a case.this case.
# Open the GitLab web interface for the repository
# In the tool bar at the top click .'''+'''. and '''New merge request''' in the drop-down menu
#:Note: you can not go via <code>Merge Requests</code> -> <code>Create merge request</code> because this will redirect to your developer fork and you can not select the master repository as source from there.
# Select the <code>dev</code> branch of the master repository as source
# Select your developer fork as target and choose branch to be updated
#::[[File:Git-fork-update-by-merge-request.png]]
# Press '''Compare branches and continue'''
# Set a title, e.g. "Updating dev branch from master repository", check once again that the source is the master repository and the target your developer fork
# Take the occasion to check what has happened since your last update and scroll through the new commits
# Press '''Submit merge request'''
# The web interface will redirect to your developer fork, the CI pipeline is started
# Once the CI succeeds, merge '''by fast-forward''' - Important '''DO NOT MERGE BY CREATING MERGE COMMITS!'''

If the option to merge by fast-forward is not displayed, this needs to be changed in the project settings of the developer fork, see [[GitLab Developer FAQ#Select Fast-forward merge]]

== Git commits ==
Changes are added to the local clone by ''committing''. First one has to define changes to be committed by ''adding'' files.

git add some_file

Once the files are marked they can be committed

git commit

This will open an editor to allow for entering a commit message. This message will be in the version control alongside with the actual changes.

=== Log message format ===
Meaningful commit messages are a powerful resource to keep track and understand the development, and will make it easier to review code changes.

There are some commonly adopted rules
# Separate subject from body with a blank line
# Limit the subject line to 50 characters
# Capitalize the subject line
# Do not end the subject line with a period
# Use the imperative mood in the subject line
# Wrap the body at 72 characters
# Use the body to explain what and why vs. how

=== Commit strategy ===
In general it is beneficial to have multiple well described commits with smaller changes instead of one commit with a whole bulk of changes. And there is some advice making it easier on the long run.

*'''Frequency:''' Make it to your habit to commit as often as possible, better more then less. Commits can be squashed later when you prepare a merge request. By committing frequently you can keep track of intermediate versions, the latest stable version, an isolated bug, to name a few.
*'''Atomicity:''' A commit should include all changes which belong together, e.g. to compile a new executable. Atomicity must be followed when commits are finally prepared for merging to master repository. Every commit in the master repository must represent a compilable snapshot
*'''Separation:''' Separate changes not belonging together and not necessary for an atomic commit. E.g. do pure formatting changes separated from functional changes.

== Handling merge requests ==
'''This section will be updated soon'''

Git allows for two merging strategies, the ''normal'' and ''fast-forward (ff)'' merging. The ''normal'' merging adds a ''merge commit''. This adds additional information, when something has been merged. In contrast to that, the latter requires to synchronize the branch to be merged via rebasing, and does not need a merge commit. This results in a linear commit history (not necessarily in time). Whether to use the one or the other depends on the specific case and taste. Many people prefer a linear commit history. As a rule of thumb one can say:
* a relatively small and timely update should be done via ff merge
* big developments outside of the main development tree should be merged with merge commits. This allows also for detailed commit messages.

=== Automatic merges ===
Gitlab offers the possibility to initiate the merge of a Merge Request (MR) directly from the Web interface. This is very easy and straight forward, however in the Gitlab Community Edition provided by UiB IT, the ''normal merge'' procedure is used. As a consequence, every Gitlab Merge Request initiated from the Web interface will include a merge commit.

=== Fast-forward merges ===
Fast-forward merges do in contrast to normal git (and Gitlab) merges not produce a separate merge commit. Gitlab Enterprise Edition offers the feature to use fast-forward merges directly from the Gitlab Web interface. UiB IT is providing Gitlab Community Edition, so we do not have this convenient feature right away.

==== Why fast-forward merges? ====
Many people consider a linear commit history as an advantage, i.e. not to have merge commits for every merge.
A fast-forward merge can be done if only the branch to be merged has additional commits on top of the common commit history. The merge simply means to forward the tip of the target branch.

==== Workflow ====
As Gitlab does not provide the functionality, the merge needs to be done outside Gitlab in a '''local clone from the master repository'''.

In the workflow below
* adjust the project name ''wpn''
* adjust the ''user''
* adjust the branch name
* you can skip the cloning if you have a already a clone

Workflow for merging the dev brach from a user fork

'''Recipe going to be updated soon'''


Gitlab will '''automatically''' recognize that the merge request was merged manually and will '''close''' it.

GitLab best practice

2021-02-23T11:37:37Z

Mri083: /* Update fork using merge request */

[[Main Page]] -> [[Documentation]] -> [[GitLab best practice]]

This page is summarizing guidelines how to use Gitlab within the project. Some general information how to work with GitLab can be found under the following
external links:
* [https://about.gitlab.com/2016/03/08/gitlab-tutorial-its-all-connected/ Tutorial: It's all connected in GitLab]
* [https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/ GitLab workflow overview]
* [https://about.gitlab.com/2014/09/29/gitlab-flow/ What is GitLab Flow?]

GitLab is a web service for '''git''', the ''stupid content tracker'' (according to its [https://manpages.debian.org/stretch/git-man/git.1.en.html man page]).
Git implements distributed repositories, git itself does not even make an assumption about a master repository. It is however good to have such a master repository and dedicated strategy how to contribute to it.
This is, where GitLab enters the scene. The web service is hosting remote repositories and implements procedures for synchronization and content merging.

In the pCT project we have master repositories for different sub-projects. Check the repositories in the [https://git.app.uib.no/pct GitLab pCT group].

In this description, we are using the [https://git.app.uib.no/pct/pct-online pct-online] repository as an example.

== Terminology ==
When working with git/GitLab, you will meet a few terms:

* '''branch''': the tip of a sequence of commits, branches share a common history and develop independently from one point in the history
* '''merge''': two branches are brought together and the result commited with a new commit
* '''rebase''': the history of a branch is applied to another branch resulting into a new history containing the commits of both branches
* '''fork''': repository copy in a user space of GitLab service
* '''clone''': local working copy on a machine

== Default branch structure ==
The main repository has the following branches:
* '''production''': the latest production code, in this branch we have release tags
* '''master''': the latest stable release
* '''dev''': the development branch

Note that we are currently only using the '''dev''' branch. The '''production''' and '''master''' branches will be populated while we are progressing with the development.

In addition to those main branches there can be ''feature'' branches where development happens detached from the main branches. A ''feature'' branch is based on the ''dev'' branch and has a limited lifetime.

== Roles ==
There are two very different use cases, the
# '''User role''': A ''user'' wants to download the code, compile it and use it.
# '''Developer role''': A ''developer'' contributes to the development.

As a '''User''', you can download an archive from the GitLab web interface, e. g. [https://git.app.uib.no/pct/pct-online pct-online], by clicking the download symbol like in this screen shot.

[[File:Pct-online-download-button.png]]

However, even if you think you are fitting best into the user role, you might end up faster than you expect coding and changing the project files. Having content management in place right from the beginning is thus a good idea and it is recommended to use the code by creating a '''clone''' of the repository.

[[File:Pct-online-clone-button.png]]

See [[Gitlab Developer FAQ#Create repository clone]] for instructions.

== The developer fork ==
The ''developer fork'' is the main feature in Gitlab's/Github's concept of distributed development, code sharing and code review.

=== Create developer fork ===
The fork can be created from the GitLab web interface by simply clicking on the '''Fork'''-button, see [[GitLab Developer FAQ#How do I create a project fork?]] for details.

To make the development within the project more coherent, you should follow a couple of [[GitLab Developer FAQ#Which rules should I obey for working with the fork? | rules under this link]].

'''Note:''' If the master repository is not public, also your fork won't be public and you have to allow your colleagues to access it, see in the rules above.

=== Synchronizing developer fork with main repository ===
'''This should be done regularly!'''

There is unfortunately no simple synchronization feature for forks within Gitlab (from Gitlab Enterprise Edition 8.2 ''repository mirroring'' is provided). Synchronization is done in a local clone or via merge reqursts.

In fact, the matter is not so easy in itself as there can be conflicting developments in the master repository and your development fork.

The primary goal of the update is to move the head of a branch in the development fork forward in the sequence of commits, from one point in the commit history to a another one. If the branch to be updated has already new commits, meaning the commit histories have diverged, all these commits should be applied to the new point in the history. Its important to note, that no additional commits are created in this process. This procedure is called ''rebase'' as we put the commits or a branch position to a ''new base''.

==== Update fork using local clone ====
A sample sequence for updating a fork of the pct-online repository, assuming the local clone is set up already (see [[GitLab Developer FAQ#How can I make a clone of a project in my fork?]]).
# Fetch branch from the master repository
#:<pre> git fetch git@git.app.uib.no:pct/pct-online.git dev</pre>
#:or if you have configured the upstream
#:<pre> git fetch upstream dev</pre>
# Rebase local branch
#:<pre> git rebase FETCH_HEAD dev</pre>
# Push to fork
#:<pre> git push origin dev</pre>

==== Update fork using merge request ====
Another way of updating the developer fork is through a reverse merge request from the master repository to the developer fork, ''reverse'' in the sense that the flow of contributions is usually in the opposite direction.

'''Note:''' we only want fast-forward merges in this case, this can not be done if the source branch is not a direct continuation of the target branch, i.e. new commits are present. Do the rebase in the local clone in such a case.this case.
# Open the GitLab web interface for the repository
# In the tool bar at the top click .'''+'''. and '''New merge request''' in the drop-down menu
#:Note: you can not go via <code>Merge Requests</code> -> <code>Create merge request</code> because this will redirect to your developer fork and you can not select the master repository as source from there.
# Select the <code>dev</code> branch of the master repository as source
# Select your developer fork as target and choose branch to be updated
#::[[File:Git-fork-update-by-merge-request.png]]
# Press '''Compare branches and continue'''
# Set a title, e.g. "Updating dev branch from master repository", check once again that the source is the master repository and the target your developer fork
# Take the occasion to check what has happened since your last update and scroll through the new commits
# Press '''Submit merge request'''
# The web interface will redirect to your developer fork, the CI pipeline is started
# Once the CI succeeds, merge '''by fast-forward''' - Important '''DO NOT MERGE BY CREATING MERGE COMMITS!'''

If the option to merge by fast-forward is not displayed, this needs to be changed in the project settings of the developer fork, see [[GitLab Developer FAQ#Select Fast-forward merge]]

== Git commits ==
=== Log message format ===

== Handling merge requests ==
'''This section will be updated soon'''

Git allows for two merging strategies, the ''normal'' and ''fast-forward (ff)'' merging. The ''normal'' merging adds a ''merge commit''. This adds additional information, when something has been merged. In contrast to that, the latter requires to synchronize the branch to be merged via rebasing, and does not need a merge commit. This results in a linear commit history (not necessarily in time). Whether to use the one or the other depends on the specific case and taste. Many people prefer a linear commit history. As a rule of thumb one can say:
* a relatively small and timely update should be done via ff merge
* big developments outside of the main development tree should be merged with merge commits. This allows also for detailed commit messages.

=== Automatic merges ===
Gitlab offers the possibility to initiate the merge of a Merge Request (MR) directly from the Web interface. This is very easy and straight forward, however in the Gitlab Community Edition provided by UiB IT, the ''normal merge'' procedure is used. As a consequence, every Gitlab Merge Request initiated from the Web interface will include a merge commit.

=== Fast-forward merges ===
Fast-forward merges do in contrast to normal git (and Gitlab) merges not produce a separate merge commit. Gitlab Enterprise Edition offers the feature to use fast-forward merges directly from the Gitlab Web interface. UiB IT is providing Gitlab Community Edition, so we do not have this convenient feature right away.

==== Why fast-forward merges? ====
Many people consider a linear commit history as an advantage, i.e. not to have merge commits for every merge.
A fast-forward merge can be done if only the branch to be merged has additional commits on top of the common commit history. The merge simply means to forward the tip of the target branch.

==== Workflow ====
As Gitlab does not provide the functionality, the merge needs to be done outside Gitlab in a '''local clone from the master repository'''.

In the workflow below
* adjust the project name ''wpn''
* adjust the ''user''
* adjust the branch name
* you can skip the cloning if you have a already a clone

Workflow for merging the dev brach from a user fork

'''Recipe going to be updated soon'''


Gitlab will '''automatically''' recognize that the merge request was merged manually and will '''close''' it.

GitLab Developer FAQ

2021-02-23T11:35:53Z

Mri083: /* Which rules should I obey for working with the fork? */

[[Main Page]] -> [[Documentation]] -> [[GitLab Developer FAQ]]

This is a collection of frequently asked Gitlab questions for developers

Please '''edit''' this page and '''add''' your question, or send email to pCT@uib.no

== Authentication ==
In order to clone non-public repositories and do synchronization, an authentication method is required. It is recommended to use SSH keys

=== Register SSH key ===
# Login to [https://git.app.uib.no https://git.app.uib.no]
# go to '''User settings''' -> '''SSH Keys''' ([https://git.app.uib.no/-/profile/keys https://git.app.uib.no/-/profile/keys])
# paste public SSH key from your <code>.ssh</code> folder, something like <code>id_*.pub</code>
# select optionally a title and expiration date
# click Add

=== Create access token ===
# Login to [https://git.app.uib.no https://git.app.uib.no]
# go to '''User settings''' -> '''Access Tokens''' ([https://git.app.uib.no/-/profile/personal_access_tokens https://git.app.uib.no/-/profile/personal_access_tokens])
# Choose name and expiration date and scopes
# click Create personal access token
# Store the token in a safe place or configure the relevant application for accessing the repository with this token immediately

Hints:
* Create access tokens only for the scope with the minimal access permissions required for your purpose
* Keep in mind: the token is only visible in the web interface after creation, you can not get it later
* Unused tokens should be revoked as soon as possible

== Create repository clone ==
A '''repository clone''' is your work copy, it is created using the <code>git clone</code>-command.

Find the links for the repository to be cloned

[[File:Pct-online-clone-menu.png|300px]]

=== Clone using SSH ===
In order to connect to the GitLab service via ssh, an SSH key needs to be configured -> [[Gitlab Developer FAQ#Register SSH key]]

git clone git@git.app.uib.no:pct/pct-online.git

'''Note:''' this is an example for the <code>pct-online</code> repository

=== Clone using Access Token ===
An access token needs to be configured -> [[Gitlab Developer FAQ#Create access token]]

git clone https://gitlab-ci-token:your-token-number@git.app.uib.no/pct/pct-online

'''Note:''' insert ''your token number''

== Working with forks ==
=== What is a fork? ===
Every logged-in user can access the main repository, however only a small group of administrators has write access. To contribute, a user creates a '''fork''' (see [[Documentation#Creating_a_project_fork | here]]) from the repository. This is a repository copy in the GitLab system where a single developer or a group of developers have full access. The fork is created in a user space.

It's important not to be confused by the terminology. In contrast to the '''fork''' which is still a remote repository on the GitLab web service,
a local copy of the repository is required on the working machine in order to work on the project. This copy is referred to be a '''clone''' (see [[Documentation#Making_local_working_copy | here]]).

=== How do I create a project fork? ===
A ''project fork'' or ''repository fork'' is a copy of the original repository where a user or a group of users has/have full control. All development in our project is carried out in the individual forks. Branches of project forks are merged back into the main repository by '''merge requests''', preferably via '''fast forward''' merges. That requires developers to rebase project forks to the main repository and resolve all conflicts before requesting a merge.

See the external link [https://docs.gitlab.com/ce/gitlab-basics/fork-project.html How to fork a project] for some general introduction.

In the pCT project you can create a fork of all the main repositories in the [https://git.app.uib.no/pct GitLab pct group] by clicking the '''fork''' button.

[[File:Pct-online-fork-button.png]]

'''Note''': '''pct-online''' is an example, replace by repository you are working with.

=== Which rules should I obey for working with the fork? ===
Your repository fork is your sandbox, you can do whatever you want. Still its good to follow some rules, unless you have your own rules at hand right now, you can apply the following:
* leave the ''master'' branch in sync with the main repository, do not make commits to it
* commit your changes to the ''dev'' branch
* if you start a new feature, and it's expected to take a while, make a ''feature'' branch, e.g. ''dev-feature'' and give the feature a name
* synchronize regularly to the main repository by rebasing (tutorial coming soon)
* add your colleagues as developers to share the code with them. To simply allow read access for the project members add group '''pct''' with ''Reporter''-role.

:: [[File:GitLab-fork-members-illustration.png | 1200px]]

* select '''Fast-forward merge''' for your fork in <code>Settings</code> -> <code>General</code> -> <code>MergeRequests</code>

:: [[File:GitLab fork-setting forward-merge.png | 800px]]

=== How can I make a clone of a project in my fork? ===
'''Note'''

* The example is using '''access via SSH'''
* refer to [[GitLab Developer FAQ#Clone SSH Key]] and [[GitLab Developer FAQ#Clone using Access Token]] for further details.
* '''User.Space''' needs to be replaced by your user space, see the link in the web interface.
* Replace '''pct-online''' by repository you are working with.

 
'''Procedure'''

git clone git@git.app.uib.no:User.Space/pct-online.git

=== I have a clone of the main repository and want to connect it now to my fork? ===
This is a typical situation. You got a clone of some repository and while using it, you started developing. Now you need to connect the clone with the fork you want to use for contributing to the project.

'''Note'''
* All the examples are using '''access via SSH''', refer to [[GitLab Developer FAQ#Clone using Access Token]] and adjust accordingly.
* '''User.Space''' needs to be replaced by your user space, see the link in the web interface.
* Replace '''pct-online''' by repository you are working with.

 
'''Procedure'''

Check the remotes of your repository
git remote -v

Should show you something like
origin git@git.app.uib.no:pct/pct-online.git (fetch)
origin git@git.app.uib.no:pct/pct-online.git (push)

Since ''origin'' should refer to your development proxy, we first rename the current ''origin'' to ''upstream'':
git remote rename origin upstream

Add the fork as the new ''origin'':
git remote add origin git@git.app.uib.no:User.Space/pct-online.git

Synchronize with the fork
git remote update

Make your branches to track the branches of your fork, e.g. for the ''dev'' branch:
git checkout dev
git branch --set-upstream-to origin/dev

Check the remotes:
git remote -v

With new result:
origin git@git.app.uib.no:User.Space/pct-online.git (fetch)
origin git@git.app.uib.no:User.Space/pct-online.git (push)
upstream git@git.app.uib.no:pct/pct-online.git (fetch)
upstream git@git.app.uib.no:pct/pct-online.git (push)

== Working with clones ==
=== How many clones can I have? ===
Since a clone is just a local copy, you can have as many as you want. This is useful if you need to do some testing in a fresh repository or on a different machine.

=== How can I change the upstream link? ===
A ''remote'' has a name in the git clone, the default name is often ''origin''

'''Note'''
* The example is using '''access via SSH'''
* '''User.Space''' needs to be replaced by your user space, see the link in the web interface.
* Replace '''pct-online''' by repository you are working with.
* If you are using a different ''name'', replace origin accordingly

 
'''Procedure'''

git remote set-url origin git@git.app.uib.no:User.Space/pct-online.git

=== Can I have more than one upstream repository ===
Yes, as a more advanced option, multiple remote repositories can be added to the clone. All remote repositories are referenced through their ''names'', e.g. ''origin'' pointing to the fork and ''upstream'' pointing to main repository. this is the recommended setup to work with both the developer fork and synchronize it to the main repository.

'''Note'''
* The example is using '''access via SSH'''
* To add the fork of a colleague, replace '''pct''' by appropriate user space
* Replace '''pct-online''' by repository you are working with.
* If you want a different name, replace upstream accordingly
* After adding the new repository you need to update the remote.

 
'''Procedure'''

git remote add upstream git@git.app.uib.no:pct/pct-online.git
git remote update upstream

=== How can I publish new development in my fork ===
All work you are doing with git is primarily within the local clone. Only by ''pushing'' updates to the remote repository you make it available for others.

Once you have added commits to e.g. the '''dev''' branch, those commits can be ''pushed'' upstream to the fork (if ''origin'' refers to the fork)
git push origin dev

Or for a feature branch
git push origin dev-cool-feature

This will create a remote branch in the remote repository if not yet there, the corresponding branch in the clone will be set up to ''track'' the remote branch.
If you have pushed earlier to the remote branch and meanwhile have changed the commit history - a fully valid procedure for development branches - the tip of the remote branch and the local branch have different commit history and you can not add additional commits by simply pushing to the remote. In this case, a ''forced'' update is needed and the remote branch is set to the commit history of the local branch. Everything not common will be lost in the remote.
git push '''-f''' origin dev

=== How can I make a backup of a branch ===
coming soon
=== How do I update a branch after fulfilled merge request? ===
=== How can I delete a branch ===

== Merge request ==
All updates to the main repository are made via ''merge requests'' (github refers to them as ''pull requests''). A merge request requires the code update to be in a mergable branch in a development fork.

Merge request are also used widely to share ''work-in-progress'' with your colleagues and for code reviews. Mark such Merge request with "'''WIP:'''".

==== Preparation ====
In order to avoid conflicts, first the fork has to be updated to the main repository. The idea behind this is that all potential conflicts can be resolved by the developer with the best knowledge about the matter, while the maintainer can simple merge ''fast-forward''.

This description expects ''origin'' pointing to the developers fork. All contributions are done to branch ''dev'', thus it must be updated with the latest version of branch ''dev'' of main repository.

# Fetch update from main repository
#: <pre> git fetch upstream dev</pre>
#:* the update is now stored in the local index, we have fetched branch ''dev'' of the main repository
#:* check the log of this latest fetch
#: <pre> git log FETCH_HEAD</pre>
#:* FETCH_HEAD is a shorthand for the head of the last branch fetched and is valid only immediately after a fetch operation
# Rebase your development to the main repository, ''rebase'' means that all the new commits are done with respect to the tip of the specified branch. There can be ''merge conflicts'' which need to be resolved with appropriate changes and commits.
#: <pre> git rebase FETCH_HEAD dev</pre>
#:* keep in mind that FETCH_HEAD is only temporarily valid, so you should do this immediately after ''fetch'' (''log'' does not change anything)
#:* replace ''dev'' with appropriate branch name if other then ''dev'' is used; it is also possible to rebase a branch with other name to the remote branch, e.g.
#: <pre> git rebase FETCH_HEAD feature-name</pre>
# Check your commits and commit messages, eventually ''squash'' (combine) or ''reword''. This advanced option is described later.
# Push to fork. As the commit history has most likely been changed, the option ''-f'' (force) has to be used
#: <pre> git push -f origin dev </pre>

The branch ''dev'' in the fork is now ready for a merge request. The branch name can be adjusted, there is nothing preventing other branch names.

==== Example workflow for a Merge request ====
# Go to your gitlab user space at [https://git.app.uib.no/User.Space] (replace ''User.Space'' appropriately).
# Find the project fork, e.g. in the list of projects associated with you from the upper main menu.
# In the line with the many columns regarding the repository, click on the "+"-symbol on the right hand side and choose "New merge request"
# Select project and branch for both source and target, and click "Compare branches and continue". '''Remember''': in almost all cases you have to merge to branch ''dev'' or other feature branch, only in very rare cases to branch ''master''
# Review the list of commits in this merge request, give it a descriptive title and description, pick an assignee
# Submit the merge request

2021-02-22T16:14:41Z

Mri083: Mri083 uploaded a new version of File:Git-fork-update-by-merge-request.png

File:Git-fork-update-by-merge-request.png

2021-02-22T16:11:58Z

Mri083:

GitLab best practice

2021-02-22T16:02:10Z

Mri083: /* The developer fork */ Clarifying recipe to update fork

[[Main Page]] -> [[Documentation]] -> [[GitLab best practice]]

This page is summarizing guidelines how to use Gitlab within the project. Some general information how to work with GitLab can be found under the following
external links:
* [https://about.gitlab.com/2016/03/08/gitlab-tutorial-its-all-connected/ Tutorial: It's all connected in GitLab]
* [https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/ GitLab workflow overview]
* [https://about.gitlab.com/2014/09/29/gitlab-flow/ What is GitLab Flow?]

GitLab is a web service for '''git''', the ''stupid content tracker'' (according to its [https://manpages.debian.org/stretch/git-man/git.1.en.html man page]).
Git implements distributed repositories, git itself does not even make an assumption about a master repository. It is however good to have such a master repository and dedicated strategy how to contribute to it.
This is, where GitLab enters the scene. The web service is hosting remote repositories and implements procedures for synchronization and content merging.

In the pCT project we have master repositories for different sub-projects. Check the repositories in the [https://git.app.uib.no/pct GitLab pCT group].

In this description, we are using the [https://git.app.uib.no/pct/pct-online pct-online] repository as an example.

== Terminology ==
When working with git/GitLab, you will meet a few terms:

* '''branch''': the tip of a sequence of commits, branches share a common history and develop independently from one point in the history
* '''merge''': two branches are brought together and the result commited with a new commit
* '''rebase''': the history of a branch is applied to another branch resulting into a new history containing the commits of both branches
* '''fork''': repository copy in a user space of GitLab server system
* '''clone''': local working copy on a machine

== Default branch structure ==
The main repository has the following branches:
* '''production''': the latest production code, in this branch we have release tags
* '''master''': the latest stable release
* '''dev''': the development branch

Note that we are currently only using the '''dev''' branch. The '''production''' and '''master''' branches will be populated while we are progressing with the development.

In addition to those main branches there can be ''feature'' branches where development happens detached from the main branches. A ''feature'' branch is based on the ''dev'' branch and has a limited lifetime.

== Roles ==
There are two very different use cases, the
# '''User role''': A ''user'' wants to download the code, compile it and use it.
# '''Developer role''': A ''developer'' contributes to the development.

From the GitLab web interface [https://git.app.uib.no/pct/pct-online pct-online], an archive can be downloaded by clicking the download symbol like in this screen shot.

[[File:Pct-online-download-button.png]]

However, even if you think you are fitting best into the user role, you might end up faster than you expect coding and changing the project files. Having content management in place right from the beginning is thus a good idea and it is recommended to use the code by creating a '''clone''' of the repository.

[[File:Pct-online-clone-button.png]]

See [[Gitlab Developer FAQ#Create repository clone]] for instructions.

== The developer fork ==
The ''developer fork'' is the main feature in Gitlabs/Githubs concept of distributed development, code sharing and code review.

=== Create developer fork ===
The fork can be created from the GitLab web interface by simply clicking on the '''Fork'''-button, see [[GitLab Developer FAQ#How do I create a project fork?]] for details.

To make the development within the project more coherent, you should follow a couple of [[GitLab Developer FAQ#Which rules should I obey for working with the fork? | rules under this link]].

'''Note:''' If the master repository is not public, also your fork won't be public and you have to allow your colleagues to access it, see in the rules above.

=== Synchronizing developer fork with main repository ===
'''This should be done regularly!'''

There is unfortunately no simple synchronization feature for forks within Gitlab (from Gitlab Enterprise Edition 8.2 ''repository mirroring'' is provided). Synchronization is done in a local clone or via merge reqursts.

In fact, the matter is not so easy in itself as there can be conflicting developments in the master repository and your development fork.

The primary goal of the update is to move the head of a branch in the development fork forward in the sequence of commits, from one point in the commit history to a another one. If the branch to be updated has already new commits, meaning the commit histories have diverged, all these commits should be applied to the new point in the history. Its important to note, that no additional commits are created in this process. This procedure is called ''rebase'' as we put the commits or a branch position to a ''new base''.

==== Update fork using local clone ====
A sample sequence for updating a fork of the pct-online repository, assuming the local clone is set up already (see [[GitLab Developer FAQ#How can I make a clone of a project in my fork?]]).
# Fetch branch from the master repository
#:<pre> git fetch git@git.app.uib.no:pct/pct-online.git dev</pre>
#:or if you have configured the upstream
#:<pre> git fetch upstream dev</pre>
# Rebase local branch
#:<pre> git rebase FETCH_HEAD dev</pre>
# Push to fork
#:<pre> git push origin dev</pre>

==== Update fork using merge request ====
Another way of updating the developer fork is through a reverse merge request from the master repository to the developer fork, ''reverse'' in the sense that the flow of contributions is usually in the opposite direction.

'''Note:''' we only want fast-forward merges in this case, this can not be done if the source branch is not a direct continuation of the target branch, i.e. new commits are present. Do the rebase in the local clone in such a case.this case.
# Open the GitLab web interface for the repository
# In the tool bar at the top click '''+''' and '''New merge request''' in the drop-down menu
#:Note: you can not go via <code>Merge Requests</code> -> <code>Create merge request</code> because this will redirect to your developer fork and you can not select the master repository as source from there.
# Select the <code>dev</code> branch of the master repository as source
# Select your developer fork as target and choose branch to be updated
# Press '''Compare and continue'''
# Set a title, e.g. "Updating dev branch from master repository", check once again that the source is the master repository and the target your developer fork
# Take the occasion to check what has happened since your last update and scroll through the new commits
# Press '''Submit merge request'''
# The web interface will redirect to your developer fork, the CI pipeline is started
# Once the CI succeeds, merge '''by fast-forward''' - Important '''DO NOT MERGE BY CREATING MERGE COMMITS!'''

If the option to merge by fast-forward is not displayed, this needs to be changed in the project settings of the developer fork.

== Git commits ==
=== Log message format ===

== Handling merge requests ==
'''This section will be updated soon'''

Git allows for two merging strategies, the ''normal'' and ''fast-forward (ff)'' merging. The ''normal'' merging adds a ''merge commit''. This adds additional information, when something has been merged. In contrast to that, the latter requires to synchronize the branch to be merged via rebasing, and does not need a merge commit. This results in a linear commit history (not necessarily in time). Whether to use the one or the other depends on the specific case and taste. Many people prefer a linear commit history. As a rule of thumb one can say:
* a relatively small and timely update should be done via ff merge
* big developments outside of the main development tree should be merged with merge commits. This allows also for detailed commit messages.

=== Automatic merges ===
Gitlab offers the possibility to initiate the merge of a Merge Request (MR) directly from the Web interface. This is very easy and straight forward, however in the Gitlab Community Edition provided by UiB IT, the ''normal merge'' procedure is used. As a consequence, every Gitlab Merge Request initiated from the Web interface will include a merge commit.

=== Fast-forward merges ===
Fast-forward merges do in contrast to normal git (and Gitlab) merges not produce a separate merge commit. Gitlab Enterprise Edition offers the feature to use fast-forward merges directly from the Gitlab Web interface. UiB IT is providing Gitlab Community Edition, so we do not have this convenient feature right away.

==== Why fast-forward merges? ====
Many people consider a linear commit history as an advantage, i.e. not to have merge commits for every merge.
A fast-forward merge can be done if only the branch to be merged has additional commits on top of the common commit history. The merge simply means to forward the tip of the target branch.

==== Workflow ====
As Gitlab does not provide the functionality, the merge needs to be done outside Gitlab in a '''local clone from the master repository'''.

In the workflow below
* adjust the project name ''wpn''
* adjust the ''user''
* adjust the branch name
* you can skip the cloning if you have a already a clone

Workflow for merging the dev brach from a user fork

'''Recipe going to be updated soon'''


Gitlab will '''automatically''' recognize that the merge request was merged manually and will '''close''' it.

GitLab best practice

2021-02-22T14:10:19Z

Mri083: small corrections