Build agent rebuilds projects which haven't had changes to their source
We just upgraded to TFS 2018 RC 2 on prem. We also installed the build agent that it suggests (version 2.120.2).
We use a branch policy to trigger builds on pull requests, and we have one build agent assigned to a pool for our builds which have big database projects. We did this because we noticed that the build agent is sometimes able to use a previous dacpac for the current build's artifacts if none of the files in the database project changed.
With the latest release of the agent, we noticed that this behavior changed, and now all of the projects build regardless of if the underlying files changed. Not only has this lengthened our build time, it has also lengthened our release time because we only deploy the database if the dacpac's hashed value changes. If the build agent creates a new dacpac unnecessarily, the hash changes and the release releases it.
In an effort to return to the previous behavior, we installed the last version of the agent that we were using (version 2.105.7). It is able to communicate with our TFS 2018 environment and it stopped rebuilding stuff unnecessarily, so we have a workaround.
I've uploaded the build logs from a release on the prior agent and one on the current agent. The one from the current agent should have skipped the builds like the one on the prior agent did.
These builds are in two different directories so they are not going to share working folders. Based on these logs I don't see any difference in behavior for the agent, just a different MSBuild log.
Can you run the builds with the System.Debug variable set to true and upload those logs.
Hi Chris, thanks for taking a look at this. The problem I'm having is build to build. So in those logs I'm not expecting the various projects to share the same artifacts (if they did, I'd have a real problem on my hands!). But the behavior we observed with the older agent is that from build to build the second build would not rebuild a project if the particular commit which caused the rebuild did not modify the underlying code in a particular project. It may take me a few more days to get a solid reproduction of this with both agents, but I will definitely upload additional logs with the System.Debug variable set to true so you can get more detail. Thanks again for taking a look!
Hi @Chris Patterson, I've attached 4 build logs.
This first file is from the older version of the agent, 21057-logs-6780-files-in-project-changed.zip, the files in the project subdirectory changed, and the build built the project appropriately. In the following build, 21057-logs-6781-files-in-project-no-change.zip , some files outside of the project subdirectory changed, and the older version of the agent did not build the project.
... I hit the attachment limit. Will hopefully post the remaining files in another comment.
With the newer version of the agent, the agent is always building all of the projects, regardless of if there is change in the code in the subdirectory. Here's a file, 21202-logs-6803-files-in-project-changed.zip, where it was appropriate to do so (for the Clarity Report project, but not the others). And again, here's a file, 21202-logs-6804-files-in-project-no-change.zip, where it shouldn't have rebuilt any of the projects.
Thank you in advance for your work on our behalf. Please let us know if you need more data to create the reproduction and identify a fix!
Hello Mark, We have the same issue after update in July the build agent on premise to version 2.120.2 (Version 1 of the agent was deprecated on August 1, 2017). Meanwhile we have version 2.124.0 and issues still exists. We use "Continuous Integration" for build every change to matching branches and we have big solution with many projects. Behavior is changed, now all (exclude native C++) of the projects build regardless of if the underlying files changed. That drastically extended build time at every push/check-in.
Steps to reproduce
1. In the build definition for the "Build Clarity Solution" step (in your case) under "Advanced" set "Create Log File".
2. Start this build once.
3. Start the same build second time.
4. The log file is on agent machine and is not included in zip file.
5. Connect to the agent where the build was performed.
6. Navigate to build directory, "C:\Source\Clarity_work\1\s" (in your case).
7. Open the log file "Clarity.sln.log" (in your case).
8. Look in this file for "is newer than output file".
9. The first line found says that build process is comparing the change time of the file "C:\Source\Clarity_temp.NETFramework, Version = v4.5.2.AssemblyAttributes.cs".
10. A few lines below the compiler "csc.exe" is called.
Possible problem cause
While upgrading the build agent to 2.x user specific %TEMP% environment variable for "msbuild" process is set on "<Agent>_temp" directory. However, this directory is completely cleaned up in the "Initialize Job" and "Post Job Cleanup" steps.
Workaround (Don’t recommend changing the original targets forever)
1. Connect to the agent where the build is running.
2. Navigate to "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\amd64".
3. Open the file "Microsoft.Common.CurrentVersion.targets" and search for "GenerateTargetFrameworkMonikerAttribute".
4. In the following node "<TargetFrameworkMonikerAssemblyAttributesPath" replace the '$([System.IO.Path]::GetTempPath())' by e.g. 'C:\MonikerTemp'.
5. Create the directory "C:\MonikerTemp".
6. Start this build once.
7. Start the same build second time.
8. If you search for "csc.exe" or "CL.exe" in the build log, no results will be found.
Thanks for posting that it is not something that had occurred to me. You shouldn't need to change the common targets and probably shouldn't. You should be able to pass that as a property as /p:TargetFrameworkMonikerAssemblyAttributesPath to MSBuild on the command line or as an environment variable.
We made the temp path change to help with other issues customers were facing around contention of multiple agents on the same box.
There is a way to override that behavior by setting an environment variable on the machine VSTS_OVERWRITE_TEMP=false and then restarting the agent.
Hello Chris, Thanks for a quick reply. Your solution "setting to environment variable on the machine VSTS_OVERWRITE_TEMP = false" works for me.
I don’t want to change original common targets forever just as workaround. I will correct my post so no one is faking my workaround.
41 People are following this
