VS2015 Update 2 VC.DB git mayhem

So today, after updating Visual Studio 2015 to Update 2, I was committing some changes to a project. Nestled amongst my own changes, deleted files, and new files, was a new VS2015 file, AOX3.VC.db (AOX3 is the name of my solution). This file is the new database engine and replaces (I believe) the SDF file. In Update 1 this was an experimental feature, but as of update 2 it’s official:

The new SQLite-based database engine is now being used by default.

I inadvertently added the file to my local git repository and then committed the changes.(I did the same thing on a couple of other projects, and made several more modifications and commits afterward).

The problem is, this file shouldn’t be in git, especially because it is around 150MB !

I only realised I had a problem when I tried to sync to my online (remote) repository, hosted by VSTS – luckily the attempt failed. The Synchronization panel had the following error message:

Failed to push to the remote repository. See the Output window for more details.

The output window had the following details:

Error encountered while pushing to the remote repository: Error while copying content to a stream.

Inner Exception:

Unable to read data from the transport connection: The connection was closed.

After a bit of googling I found I had to resort to the nuclear option to fix the problem.. deleting the file from the local git commit history.

Here’s what I did:

1. In VS2015 find the first commit that included the file. To do this I went to the Sync panel on the Team Explorer page and found the outgoing commit, then double-clicked it to see the commit details:

 

Screen Shot 2016-03-30 at 20.25.24.png

Screen Shot 2016-03-30 at 20.27.43.png

2. Note the parent commit ID of the first commit that included the new file (shown above, 645faa8c), then close VS2015.

3. Run a git bash shell in the solution folder. Then use the filter-branch command to delete the file from the last two commits. This required specifying a range of commits from the parent of the first commit all the way up the HEAD (the latest commit). This means noting the parent commit ID of the first commit that included the new file – as shown in the above screenshot. Here’s the git session:

jon@DESKTOP-0J37S2C MINGW64 /c/dev/Projects/Amber/AOX3 (dev_jon)
$ git filter-branch --tree-filter 'rm -f AOX3.VC.db' 645faa8c..HEAD
Rewrite 2fa87507296744092c2ac8b0c300b8c5973276ef (1/2) (0 seconds passed, remain
Rewrite 2917e9beea5ec0f546538345e33bd01586d359fb (2/2) (15 seconds passed, remaining 0 predicted)
Ref 'refs/heads/dev_jon' was rewritten

4. Open VS2015 and check the commits.. now the added/modified database file has disappeared from the list of changes:

Screen Shot 2016-03-30 at 20.35.52.png

5. I then edited my .gitignore file and added *.VC.db, then committed this mod and finally checked everything synchronized.. all worked as expected !

 

I guess the moral of the story is: don’t be flippant when adding new files, especially those you don’t recognize as your own. As the filter-branch documentation states:

This occurs fairly commonly. Someone accidentally commits a huge binary file with a thoughtless git add ., and you want to remove it everywhere.

This thoughtless programmer signing off, time to sleep😉

 

 

OpenCV3, VS2015

logo

Intro

So I finally got myself a clean Windows 10 VM for the next couple of years development, installed Visual Studio 2015, downloaded OpenCV, migrated my VS2013 OpenCV C++ test app, and.. and.. of course it didn’t work😦

This was only an issue because I was using the static OpenCV libraries rather than the pre-built binaries. It looks like VS2015 is quite happy using the VC12 OpenCV binaries.

The current release of OpenCV 3.0.0 (04/06/2015) comes with pre-built static and import libraries for VS2013 and VS2010, but not for VS2015. I found an excellent article on the Colorado School of Mines (!) WebSite on how to build VS2015 libraries. But I needed some other tweaks beyond a basic build of OpenCV. So the goal of this note is to show how I built OpenCV for use with Visual Studio 2015 with the following configurations:

  1. 32- and 64-bit support.
  2. Static and Shared OpenCV libraries.
  3. Debuggable static libraries by having the correct combination of source code, libraries and PDBs.

The basic procedure I’m using (after several iterations) is:

  • Clone version 3.0.0 of OpenCV to a local development folder. I’m going to keep these source files handy because the debugger can use them in Debug builds
  • Install CMake, a build configuration tool.
  • Generate 4 VS2015 solutions
    • x86, static
    • x86, shared
    • x64, static
    • x64, shared

Clone OpenCV

This assumes you have VS2015 and the latest Git for Windows.

The trick, at least for me, is to clone into a folder that will be used to keep the OpenCV source code. This folder won’t get deleted during the steps below. I’m also only interested in the official latest release, which at the time of writing is version 3.0.0.

I have a standard development folder structure which is currently based on c:\dev, and I have a standard sub-folder structure for all third party libraries, c:\dev\ThirdParty. So I’m going to put OpenCV into c:\dev\ThirdParty\OpenCV3.0.0.

Run a git bash shell (from any folder) and clone version 3.0.0. Note the use of forward slashes in both paths !

Here’s my bash session:
jon@DESKTOP-0J37S2C MINGW64 /c/dev/ThirdParty/OpenCV3.0.0 ((3.0.0))
$ git clone --branch 3.0.0 https://github.com/Itseez/opencv.git C:/dev/ThirdParty/OpenCV3.0.0/sources
Cloning into 'C:/dev/ThirdParty/OpenCV3.0.0/sources'...
remote: Counting objects: 183285, done.
remote: Compressing objects: 100% (52/52), done.
remote: Total 183285 (delta 22), reused 6 (delta 6), pack-reused 183224
Receiving objects: 100% (183285/183285), 419.28 MiB | 3.54 MiB/s, done.
Resolving deltas: 100% (124890/124890), done.
Checking connectivity... done.
Note: checking out 'c12243cf4fccf5df7b0270a32883986b373dca7b'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

Checking out files: 100% (4656/4656), done.
Now I have the following folder structure:
471B7A0B-A24A-43FB-966C-14CB3E80C70F.png

Get CMake

There isn’t a VS2015 solution bundled with the OpenCV download so you must create one. This is made easy by using CMake, a cross-platform build utility. Download and install from here – I used the Windows (Win32 Installer) ‘cmake-3.4.1-win32-x86.exe’, and kept the default installation options.

Create the VS2015 Solutions

The basic procedure with CMake is:

  • Set the source and build folders
  • Click Configure
  • Choose the generator (compilers)
  • Tweak the options
  • Click Generate

I’m going to place each solution in a unique build folder and will start with:

  • 32-bit (x86)
  • Shared OpenCV_World300.dll
  • Static C runtime (CRT)
  • No test libraries (because they don’t build cleanly for me, and I want a clean build for the moment)

Run CMake and set the source code folder. Then choose a folder for the build. Make sure the build folder doesn’t already exist. E.g.

W10_Dev.png

The click Configure, accept any request to create the build folder, and then choose the generator for the project. For x86:

W10_Dev.png

The first pass at the configuration takes a minute or so and may download some IPPI files. It also ends up looking quite bad but the red items are just new values (which initially means everything):

W10_Dev.png

Use the Search box to find and verify/update the following settings:

  • BUILD_opencv_world: True
  • BUILD_SHARED_LIBS: True
  • BUILD_WITH_STATIC_CRT: True
  • BUILD_PERF_TESTS, BUILD_TESTS: False

For example, to find and update the OpenCV_World option:

W10_Dev.png

 

Click Generate. Then check you have a new VS2015 solution (but don’t open or build yet):

  • C:\dev\ThirdParty\OpenCV3.0.0\build_x86_shared\OpenCV.sln

For the remaining 3 VS2015 solutions first change the build path, click Configure, apply the settings, and click Generate.

  • x86, static
    • Build path: C:/dev/ThirdParty/OpenCV3.0.0/build_x86_static
    • Generator: Visual Studio 14 2015
    • BUILD_opencv_world: False
    • BUILD_SHARED_LIBS: False
    • BUILD_WITH_STATIC_CRT: False
    • BUILD_PERF_TESTS, BUILD_TESTS: False

 

  • x64, shared
    • Build path: C:/dev/ThirdParty/OpenCV3.0.0/build_x64_shared
    • Generator: Visual Studio 14 2015 Win64
    • BUILD_opencv_world: True
    • BUILD_SHARED_LIBS: True
    • BUILD_WITH_STATIC_CRT: True
    • BUILD_PERF_TESTS, BUILD_TESTS: False

 

  • x64, static
    • Build path: C:/dev/ThirdParty/OpenCV3.0.0/build_x64_static
    • Generator: Visual Studio 14 2015 Win64
    • BUILD_opencv_world: False
    • BUILD_SHARED_LIBS: False
    • BUILD_WITH_STATIC_CRT: False
    • BUILD_PERF_TESTS, BUILD_TESTS: False

 

Build x86 static

Close CMake and open VS2015, then open the x86 static solution:

  • C:\dev\ThirdParty\OpenCV3.0.0\build_x86_shared\OpenCV.sln

Give it a minute or two for the parsing stage to complete, select the Debug|Win32 configuration, and build the solution.

W10_Dev.png

I got the following output:

Build: 14 succeeded, 0 failed, 0 up-to-date, 5 skipped

Next build the CMakeTargets/INSTALL project – this will put the output files in an install folder which we’ll use later on:

W10_Dev.png

You can check you have the opencv_world300 debug DLL:

W10_Dev_and_Open.png

Now switch to the Release|Win32 configuration, build the solution, and build the CMakeTargets/INSTALL project. This will add the opencv_world release DLL to the install folder.

Now open the x64 shared solution and repeat the above to get the debug and release opencv_world300 DLLs. Note the configurations are:

  • Debug|x64
  • Release|x64

 

Open the x86 static solution: before build we must fix a glitch with the PDB file names. For some reason the debug PDBs were not configured to have a d appended to the filename. This means the debug and release PDBs will all use the same filenames. For example:

  • IlmImf.lib -> IlmImf.pdb
  • IlmImfd.lib -> IlmImf.pdb

There is probably a quick way to fix this using the CMake tools, but I don’t know how to ! Instead, do the following:

  1. Multi-select all the projects in the 3rdparty groupW10_Dev.png
  2. Open the Property Pages (right-click and select Properties).
  3. Set the Configuration to ‘All Configurations’
  4. Navigate in the tree-view to C/C++ / General
  5. Change the ‘Debug Information Format’ to ‘C7 compatible’
    W10_Dev.png
  6. Hit OK to apply the changes and close the Property Pages.

Repeat the above procedure for all projects in the following groups:

  • applications
  • modules
  • object_libraries

Why do this ?

  1. Instead of producing .lib and .pdb files we now have the PDB information merged into the .lib.
  2. It solves the problem of the debug build PDBs not having different names from the release equivalents.
  3. The CMakeTargets/INSTALL project doesn’t copy the PDBs, meaning a further manual step would be required to find the PDBs and copy them to the install folder.

Is there a better way – I’m sure there is, but this is quite simple and quick and if it works then why not ? (One good reason to not use this approach is if you’re going to work on the OpenCV source code; using the C7 option takes away certain compiler and linker options that could make for much faster builds). There’s plenty of discussion on the web about this, for example on StackOverflow.

Repeat the previous build procedure to get the Debug and Release builds and INSTALLs done. Then repeat again for the x64 (remembering to modify the Debug Information Format settings first).

Consolidating the build

I want to replicate the folder structure of the official release of OpenCV as much as possible. So first, create a new build folder:

  • C:\dev\ThirdParty\OpenCV3.0.0\build

Then copy the contents of each of the install folders into the above build folder. Skip over duplicate files.

  1. build_x64_shared
  2. build_x64_static
  3. build_x86_shared
  4. build_x86_static

My final folder structure is around 1.2GB and looks like this

 

W10_Dev.png

All four of the build_XXX folders can now be deleted – for me they take up around 10GB. Alternatively you can reload each solution, select Batch Build, Select All, then clean. This drops the folders down to around 2GB. For me I’m going to ZIP and archive them, in case I made a mistake !

Testing the build

One of my requirements is to make sure colleagues can work with these libraries, even if they use a different folder structure to me for their own projects. The only prerequisite is that there’s a ThirdParty\OpenCV3.0.0 folder; but ThirdParty can be anywhere.

My first trick is to make a User Macro in VS2015 to set the ThirdParty folder location. This is another example of something that could probably be done better, but I don’t want to change the system PATH and I need something simple. My reference for User Macros is from this MSDN page.

  1. Open or create a C++ project.
  2. View the Property Manager:
    W10_Dev.png
  3. Select the Microsoft.Cpp.Win32.user node of any project, right-click and select Properties. Then select the User Macros setting and add a Macro called ThirdParty:
    W10_Dev.png
  4. Apply the changes and close the dialog, then close the Property Manager.

The macro can now be used in projects which may be distributed among other developers; each developer just needs to have their ThirdParty folder (with OpenCV3.0.0 inside) and this macro.

As an example, I have a 32-bit project using the static OpenCV library. The key settings are:

Additional Include Directories:W10_Dev.png

Additional Library Directories:W10_Dev.png

Additional Dependencies:W10_Dev.png

The list of dependencies for my Release build is:

IlmImf.lib
libjasper.lib
libjpeg.lib
libpng.lib
libtiff.lib
libwebp.lib
opencv_calib3d300.lib
opencv_core300.lib
opencv_features2d300.lib
opencv_flann300.lib
opencv_hal300.lib
opencv_highgui300.lib
opencv_imgcodecs300.lib
opencv_imgproc300.lib
opencv_ml300.lib
opencv_objdetect300.lib
opencv_photo300.lib
opencv_shape300.lib
opencv_stitching300.lib
opencv_superres300.lib
opencv_ts300.lib
opencv_video300.lib
opencv_videoio300.lib
opencv_videostab300.lib
zlib.lib
ippicvmt.lib

And for my debug build it is:

IlmImfd.lib
libjasperd.lib
libjpegd.lib
libpngd.lib
libtiffd.lib
libwebpd.lib
opencv_calib3d300d.lib
opencv_core300d.lib
opencv_features2d300d.lib
opencv_flann300d.lib
opencv_hal300d.lib
opencv_highgui300d.lib
opencv_imgcodecs300d.lib
opencv_imgproc300d.lib
opencv_ml300d.lib
opencv_objdetect300d.lib
opencv_photo300d.lib
opencv_shape300d.lib
opencv_stitching300d.lib
opencv_superres300d.lib
opencv_ts300d.lib
opencv_video300d.lib
opencv_videoio300d.lib
opencv_videostab300d.lib
zlibd.lib
ippicvmt.lib

To make sure it’s all looking good I build the Debug configuration and check there are no spurious errors or warnings. (For example, before I switched to the C7 debug-info setting I had all sorts of warnings about missing PDBs.)

Then try setting a breakpoint on some OpenCV code and stepping into it: if all goes well you’ll be able to step into and around the OpenCV source code.

Note: if you move your OpenCV distribution to another folder then the debugger will prompt you for the location of the OpenCV source files. This only happens once (as far as I can tell).

To use the DLL version of OpenCV change the following:

  • Linker/General/Additional Library Directories:
    $(ThirdParty)\OpenCV3.0.0\build\x86\vc14\lib;%(AdditionalLibraryDirectories)
  • Linker/Input/Additional Dependencies:
    opencv_world300.lib

Note: if you run a shared DLL version without making the DLL available then you’ll get something like this:

‘The program can’t start because opencv_world300.dll is missing from your computer.’

W10_Dev.png

To avoid changing the system PATH I added a post-build event to copy the DLL to my project’s output:

W10_Dev.png

A good sample program to start with is Load and Display an Image, on the OpenCV website.

That’s it – I’m done ! After at least 10 variations of these builds, and two or three attempts to write up how I did it, I’m leaving it here.  At least until I figure a better way to do it, or an update is issued from OpenCV, or I figure out how to try and contribute directly to OpenCV, etc.  :-)

 

Visual Studio 2013 to 2015, MSB3884, Could not find rule set file

After loading a VS2013 C# project in VS2015 I got the dreaded MSB3884 warning (Could not find rule set file). A little annoying since I wasn’t knowingly using code analysis (but I might now).

Here’s what I did to fix the warning:

  1. On the project properties go to the Code Analysis section.
  2. Select the Browse option on the list of rule sets:
    W10_Test.png
  3. Browse to the VS2015 rule sets folder:
    C:\Program Files (x86)\Microsoft Visual Studio 14.0\Team Tools\Static Analysis Tools\Rule Sets
  4. Pick a rule set.. the default used by new projects is:
    MinimumRecommendedRules.ruleset
  5. The new rule set is applied:
    W10_Test.png

Then rebuild – problems gone !

 

 

Python 2.7 stack, OpenCV 3, Visual Studio 2013

I’ve started working with Python (2.7 and 3.x) and OpenCV 3 recently, and the IPython Notebook way of working with inline plots is awesome. But the debugging, editing and some other features are not great when compared to working in a Visual Studio environment.

So now I’m trying Visual Studio 2013 as the IDE, OpenCV 3, and Python 2.7 via the numerical and scientific WinPython stack.

I extracted OpenCV and WinPython to:

  • My Documents\OpenCV
  • My Documents\WinPython-32bit-2.7.10.2

From VS2013 I then installed Python Tools for Visual Studio (PTVS).. I did this within VS2013 using the Extensions and Tools dialog.

Finally I configured a new Python Environment:

W10_Dev

From an Interactive Window I can then check everything works as expected. First, Open CV3:

Python interactive window. Type $help for a list of commands.
import cv2
cv2.__version__
'3.0.0'

Next up, NumPy and a MatPlotLib chart:

import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(0, np.pi * 2, 100)
y = np.sin(x)
plt.plot(x, y)
plt.xlabel('Radians')
plt.ylabel('sin(x)')
plt.show()

This gives us a popup (model) window:

W10_Dev

Now to make it a bit more interesting – inline plots.. ! The Python Interactive Window has options for the Interactive mode:

W10_Dev

The default Interactive Mode is Standard but we can change this to IPython:

W10_Dev

After restarting VS2013 we can now go inline…

W10_Dev

And lastly an OpenCV 3 test:

In [4]: import cv2
In [5]: img = cv2.imread(r"C:\Users\jon\Pictures\Samples\eye.jpg", 0)
In [6]: plt.imshow(img, cmap='gray')
Out[6]: <matplotlib.image.AxesImage at 0x5b8e510>

W10_Dev

In [34]: blur = cv2.medianBlur(img, 91)
In [35]: plt.imshow(blur, cmap='hot')
Out[35]: <matplotlib.image.AxesImage at 0x1edea7d0>

W10_Dev

The great thing is I can now move this code into a Python project, get it into git, distribute among colleagues, and use the excellent intellisense and debugging powers of PTVS ! (Obviously not this actual code !)

JSON editing in Visual Studio 2013 – tip!

Just installed VS2013 on a clean Windows 10 virtual machine, noticed I wasn’t getting the same JSON editing experience I had on my previous VS2013 installation. The colour coding and auto-completion features were missing.

Solved the problem by modifying the installation and including the Microsoft Web Developer Tools:

W10_Dev

Now the editor uses colour coding, auto-completes the various brackets, and highlights errors.

OS X: IPython Notebooks on an an external drive with Anaconda

Anaconda, running on El Capitan, only provides access to files and folders under the current user’s home directory.

Home

My Dropbox folder lives on an external drive and therefore isn’t accessible from the folder list. (I actually use a slimline 128 GB JetDrive card from Transcend.. fast enough for my 100+ GB of Dropbox files).

The solution was to create a symbolic link in my Documents folder which referenced the external Dropbox folder.

$ ln -s /Volumes/SDCARD/Dropbox ~/Documents/DB

From Anaconda’s perspective this looks like any other folder:

Documents_DB_