using GYP/depot_tools and building pdfium

foreword & couple words of GYP

Recently I've needed to build pdfium - that engine, which is rendering pdf in chrome browser. So, I've found these links:

• 'Building PDFium' instruction [link]

and on this page was link to 'depot_tools'

• 'Install depot_tools' instruction [link]

So, what's that. Google fellows made great universal cross-platform building system - GYP (generalize your project). It's like cmake, and for cases with I encounted - I liked it much more that cmake (my subjective opinion). 'depot_tools' - it is GYP.



problem

And here was a problem - 'Non-cygwin' part of documentation is obsolete and if you make everything what documentation says - nothing will work. And even if you succsseed in documentation part - you will get a lot of not described problems.
In this note I will describe solving this problems.


tools & versions

I have installed on my pc:

• Windows 8.1
• Visual Studio 2013 professional
• git version 1.9.4.msysgit.0
• All filesystem/archiving operations I'm doing with FAR Manager 3.0

problem solving in a short

0. Create 'c:\depot_tools' directory (or any another you want) & add it to %PATH%

1. Forget about archive with tools in documentation - take last version from git:
    >cd c:\
    >git clone https://chromium.googlesource.com/chromium/tools/depot_tools
    >cd ./depot_tools

2. Make some git magic (without it nothing will work)
    >git reset --hard HEAD

3. run 'gclient'
    When it asks you
        Your depot_tools checkout is configured to fetch from an obsolete URL
        Would you like to update it? [y/N]:
    answer Y
    If everything is ok - you will get something like that, without any errors:

4. This step probably optional. Your python - 'C:\depot_tools\python276_bin' doesn't have gyp module. Last time it generated errors, but not now. I don't know why. So, if you have errors bcs of absent 'gyp' module or just wanna have it 'just in case' - install it.

    somewhere in temp directory:
    >git clone https://chromium.googlesource.com/external/gyp
    >C:\depot_tools\python276_bin\python.exe setup.py install

And here you have working 'depot_tools'. And next steps for those who wants to build pdfium (as bonus, you will get ready 'v8' engine as dependency).

5. Somewhere where you want to put pdfium sources (for example in "D:\tmp"), exec next commands (it will download ~600 Mbytes of sources):
    >mkdir pdfium
    >cd pdfium
    >gclient config --name . --unmanaged https://pdfium.googlesource.com/pdfium.git
    >gclient sync

Now you have project pdfium.

6. To build .sln files exec:
    >C:\depot_tools\python276_bin\python.exe D:\tmp\pdfium\build\gyp_pdfium

    You will see something like that:
--------------------------------------
Updating projects from gyp files...
Warning: Missing input files:
D:\tmp\pdfium\testing\gtest-support.h
D:\tmp\pdfium\testing\gmock_mutant.
-------------------------------------- 
    if it so - everything is fine. Everything will build even without these files.

7. During building process, v8 engine expects to find python in 'D:\tmp\pdfium\v8\third_party\python_26'. So, create this directory and copy to it content of your 'C:\depot_tools\python276_bin'

8. If you will try to build pdfium for x86 platform and see something like that:
        LINK : warning LNK4068: /MACHINE not specified; defaulting to X64
    find 'pdfium' in solutions list, go to
    pdfium -> configuration properties -> librarian -> all options
    find 'Target machine' option with empty value
    set (from dropdown):
        MachineX86 (/MACHINE:X86)
    after that on my machive I have:
        ========== Build: 37 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

9. For building for x86-64 target platform you need to generate new .sln files:
    >C:\depot_tools\python276_bin\python.exe D:\tmp\pdfium\build\gyp_pdfium -Dtarget_arch=x64
    then it builds easily


illustration of problems with following obsoleted instructions

[this part mostly for people, googling solution by error strings]
[it's illustration how you don't need to do - how you need to do you can find in part 'problem solving in a short']

Let's try follow instruction - I will follow 'non-Cygwin' branch of instruction:

1. Get a copy of depot_tools:

This 'install-depot-tools' link has really detailed instruction and zip archive with 'depot_tools' for downloading. So, download archive 'depot_tools.zip' and extract it, for exaple, to 'C:\depot_tools' (for exaple, with FAR manager)

2. Add depot_tools to the end (not start!) of your PATH ('With Administrator access' branch)

3. Non-cygwin: Run gclient from the cmd shell
so, enter in FAR 'gclient' (it will start c:\depot_tools\gclient.bat) and wait.

>gclient
Installing python 2.7.6...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/python276_bin.zip
Installing git 1.9.0.chromium.5 (avg 1-2 min download) ...
Fetching from https://commondatastorage.googleapis.com/chrome-infra/git-1.9.0.chromium.5_bin.zip
""c:\depot_tools\git.bat"" is not recognized as an internal or external command
""c:\depot_tools\git.bat"" is not recognized as an internal or external command

so, we have the same problem which that guy - link.
and you dive into this problem and find solution like described here - link.

'C:\depot_tools\bootstrap\win\win_tools.bat' from 'depot_tools.zip' contain next lines:
------------------------------------------------------------------------------------------------------
:: Ensure autocrlf and filemode are set correctly.
call "%WIN_TOOLS_ROOT_DIR%\git.bat" config --system core.autocrlf false
call "%WIN_TOOLS_ROOT_DIR%\git.bat" config --system core.filemode false
goto :GIT_COPY_BATCH_FILES


:GIT_COPY_BATCH_FILES
:: Create the batch files.
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\git.bat" "%WIN_TOOLS_ROOT_DIR%\git.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\gitk.bat" "%WIN_TOOLS_ROOT_DIR%\gitk.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\ssh.bat" "%WIN_TOOLS_ROOT_DIR%\ssh.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\ssh-keygen.bat" "%WIN_TOOLS_ROOT_DIR%\ssh-keygen.bat" 1>nul
goto :SVN_CHECK
------------------------------------------------------------------------------------------------------

Firsly it's trying to call git.bat, and only after that create this file. After exec 'gclient' this file will be replaced by new version where this fixed, but when you see this error - executed old, bugged version of this file.
So, you decided to make something like that - unpack 'depot_tools' in 'C:\depot_tools' one more time and replace wrong lines by correct ones:

------------------------------------------------------------------------------------------------------
del "%ZIP_DIR%\git.zip"
goto :GIT_COPY_BATCH_FILES


:GIT_COPY_BATCH_FILES
:: Create the batch files.
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\git.bat" "%WIN_TOOLS_ROOT_DIR%\git.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\gitk.bat" "%WIN_TOOLS_ROOT_DIR%\gitk.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\ssh.bat" "%WIN_TOOLS_ROOT_DIR%\ssh.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\ssh-keygen.bat" "%WIN_TOOLS_ROOT_DIR%\ssh-keygen.bat" 1>nul
:: Ensure autocrlf and filemode are set correctly.
call "%WIN_TOOLS_ROOT_DIR%\git.bat" config --system core.autocrlf false
call "%WIN_TOOLS_ROOT_DIR%\git.bat" config --system core.filemode false
goto :SVN_CHECK
------------------------------------------------------------------------------------------------------

so, after that you will get next log:

------------------------------------------------------------------------------------------------------
C:\depot_tools>gclient
Installing python 2.7.6...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/python276_bin.zip
Installing git 1.9.0.chromium.5 (avg 1-2 min download) ...
Fetching from https://commondatastorage.googleapis.com/chrome-infra/git-1.9.0.chromium.5_bin.zip
Installing subversion ...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/svn_bin.zip
Cannot rebase: You have unstaged changes.
Please commit or stash them.
Failed to update depot_tools.
Usage: gclient.py <command> [options]

Meta checkout manager supporting both Subversion and GIT.

Commands are:
  cleanup  cleans up all working copies
  config   creates a .gclient file in the current directory
  diff     displays local diff for every dependencies
  fetch    fetches upstream commits for all modules
  grep     greps through git repos managed by gclient
  help     prints list of commands or help for a specific command
  hookinfo outputs the hooks that would be run by `gclient runhooks`
  pack     generates a patch which can be applied at the root of the tree
  recurse  operates [command args ...] on all the dependencies
  revert   reverts all modifications in every dependencies
  revinfo  outputs revision info mapping for the client and its dependencies
  runhooks runs hooks for files that have been modified in the local working copy
  status   shows modification status for every dependencies
  sync     checkout/update all modules

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -j JOBS, --jobs=JOBS  Specify how many SCM commands can run in parallel;
                        defaults to 8 on this machine
  -v, --verbose         Produces additional output for diagnostics. Can be
                        used up to three times for more logging info.
  --gclientfile=CONFIG_FILENAME
                        Specify an alternate .gclient file
  --spec=SPEC           create a gclient file containing the provided string.
                        Due to Cygwin/Python brokenness, it can't contain any
                        newlines.
  --no-nag-max          Ignored for backwards compatibility.
------------------------------------------------------------------------------------------------------

what the fuck?
    Cannot rebase: You have unstaged changes.
    Please commit or stash them.
    Failed to update depot_tools.

so, you start search solution, find links:

• https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/4Dcn2ZKjuNI
• https://groups.google.com/a/chromium.org/forum/#!topic/chromium-dev/ApAhxyiN4Kg

and you find in 'update_depot_tools.bat' line:
    call git config remote.origin.fetch > NUL

and you repeat everything one more time - unpack archive, fix lines in 'win_tools.bat', then add line to 'update_depot_tools.bat' to make next way:
    call git reset --hard HEAD > NUL
    call git config remote.origin.fetch > NUL

and finally you get this problem - https://code.google.com/p/chromium/issues/detail?id=448981

------------------------------------------------------------------------------------------------------
Installing python 2.7.6...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/python276_bin.zip
Installing git 1.9.0.chromium.5 (avg 1-2 min download) ...
Fetching from https://commondatastorage.googleapis.com/chrome-infra/git-1.9.0.chromium.5_bin.zip
'"Z:\Projects\Chromium\depot_tools\git.bat"' is not recognized as an internal or external command,
operable program or batch file.
'"Z:\Projects\Chromium\depot_tools\git.bat"' is not recognized as an internal or external command,
operable program or batch file.
Installing subversion ...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/svn_bin.zip
'*' is not recognized as an internal or external command,
operable program or batch file.
------------------------------------------------------------------------------------------------------

and on that moment I tired from this shit and searched another way to install it. And here are some more error messages, which appearing when you are dealing with GYP, depot_tools & pdfium.

------------------------------------------------------------------------------------------------------
Installing python 2.7.6...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/python276_bin.zip
Installing git 1.9.5.chromium.6 (avg 1-2 min download) ...
Fetching from https://commondatastorage.googleapis.com/chrome-infra/git-1.9.5.chromium.6_bin.zip
[-] XMLHTTP 80072ee2: Cannot make HTTP request (The operation timed out

... Failed to checkout git automatically.
You should get the "prebaked" version used at https://commondatastorage.googleapis.com/chrome-infra/git-1.9.5.chromium.6_bin.zip
------------------------------------------------------------------------------------------------------
what to do: you forgot to make 'git reset --hard HEAD' - so, step 2 in tutorial above.

during building pdfium you can find next error:
------------------------------------------------------------------------------------------------------
1>  bash: python: command not found
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets(170,5): error MSB6006: "cmd.exe" exited with code 127.
------------------------------------------------------------------------------------------------------
what to do - it can't find python in pdfium - so, step 7 in tutorial above.

if you are trying to build x64 target platform from x32 sources:
------------------------------------------------------------------------------------------------------
1>cl : Command line warning D9002: ignoring unknown option '/arch:SSE2'
1>D:\tmp\pdfium\v8\src/base/build_config.h(115): fatal error C1189: #error :  Target architecture ia32 is only supported on ia32 host (..\..\src\assert-scope.cc)
------------------------------------------------------------------------------------------------------
what to do - you need to generate another visual studio files - so, step 9 in tutorial above.